On Mon, Jul 08, 2002 at 02:44:25PM -0400, Tim Peters wrote:

[David Abrahams]

...

I keep running into the problem that there is no reliable way to introspect about whether a type supports multi-pass iterability (in the sense that an input stream might support only a single pass, but a list supports multiple passes). I suppose you could check for __getitem__, but that wouldn't cover linked lists, for example.

Can anyone channel Guido's intent for me? Is this an oversight or a deliberate design decision? Is there an interface for checking multi-pass-ability that I've missed?

The language makes no such distinctions. If an app wants to make them, it's up to the app to implement them. Likewise for a way to tell a multipass iterator to "start over again". The Python iteration protocol has only two methods, .next() to get "the next" item, and .iter() to return self; given a random iterator, those are the only things you can rely on.

I believe that when David was talking about multi-pass iterability he wasn't referring to an iterator that can be told to "start over again" but to an iterable object that can produce multiple independent iterators of itself, each one good for a single iteration. The language does make a distinction between an *iterable* object that may have only an __iter__ method and an *iterator* that has a next method. This distinction is blurred a bit by the fact that iterators also have an __iter__ method that also makes them appear as one-shot iterables. Imagine an altenative universe where a south african programmer called Rossu van Guidom writes a wonderful language called Mamba and in that language iterator semantics are defined like this: * Objects that wish to be iterable define an __iter__() method returning an iterator. * An iterator is an object with a next() method. That's all. * The for statement checks if an object has an __iter__ method. If it does, it calls it and uses the returned iterator. If it doesn't, it tries to use the object itself. If it doesn't have .next either it will fail and report that the object is not iterable. A Mamba programmer called Nero Hsorit has speculated in a mamba-dev posting that in an alternative universe in a language called 'Cobra' people kept getting confused between iterators and iterables :-) Oren

Imagine an altenative universe where a south african programmer called Rossu van Guidom writes a wonderful language called Mamba and in that language iterator semantics are defined like this:

* Objects that wish to be iterable define an __iter__() method returning an iterator.

* An iterator is an object with a next() method. That's all.

But that doesn't allow for things like file objects, which, although not iterators themselves, are capable of producing iterators of different sorts which iterate over them in different ways -- and yet they can only be iterated over once. In other words, there are such things as one-shot iterables, even if iterables and iterators are kept separate. Maybe a one-shot iterable should raise an exception if you try to obtain a second iterator from it? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

Then you couldn't do this:

done = False for line in f: if not check(line): break process(line) else: done = True

if not done: for line in file: another_process(line)

That's already broken, see SF bug 524804. --Guido van Rossum (home page: http://www.python.org/~guido/)

Oren Tirosh wrote:

Xreadlines is buffered and therefore leaves the file position of the file in an unexpected state. If you use xreadlines explicitly you should expect that. The fact that file.__iter__ returns an xreadlines object implicitly is therefore a bit surprising.

What's the reason for using xreadlines as a file iterator? Was it performance or was it just the easiest way to implement it using an existing object?

The rationale was something like "the simple most way to iterate over the lines in a file should be the fastest". I'd agree with that, but not at the expense of the surprises mentioned in the bug. I would perhaps help if the file object would cache the xreadlines iterator, that would limit the scope of the problem to the case where iteration and explicit .read() calls are mixed.

"Files support the iterator protocol. Each iteration returns the same result as file.readline()"

This is not correct. Files support what I call the iterable protocol. Objects supporting the iterator protocol have a .next() method, files don't. While it's true that each iteration has the same result as readline it doesn't have the same side effects.

Proposal: make files really support the iterator protocol. __iter__ would return self and next() would call readline and raise StopIteration if ''. If anyone wants the xreadline performance improvement it should be explicit.

+1 (But, since the bug is closed as "won't fix" I doubt this has a big chance of happening.) Just

On Thursday 11 July 2002 12:47 pm, Guido van Rossum wrote:

...

...

Proposal: make files really support the iterator protocol. __iter__ would return self and next() would call readline and raise StopIteration if ''. If anyone wants the xreadline performance improvement it should be explicit.

No. I won't have "for line in file" be slower than attainable.

+1. I _intensely_ want to be able to teach beginners to use "for line in file" and have it be fast in the common case. "Nice" behavior for rarer cases of prematurely interrupted loops is OK, if feasible, but secondary. Having "for line in file" play nicely with other method calls on 'file' has no importance to me in this context -- no more than, e.g., having "for item in alist" play nicely with calls to mutating methods of object alist.

The only solution I accept is a complete rewrite of the I/O system without using stdio, so xreadlines can be integrated.

I thought Just's suggestion (about having the file object remember the xreadlines object in use, so that another for loop would continue right where the first one exited) seemed like a reasonable hack -- a compromise of reasonably little effort for some small secondary gain. Guess I must be missing something? Of course the "complete rewrite" is an alluring prospect -- for many other reasons, such as enabling user control of file buffering in cross-platform ways, *yum* -- but it's not going to happen in time for 2.3 anyway, is it? Alex

From: "Oren Tirosh" <oren-py-d@hishome.net>

I believe that when David was talking about multi-pass iterability he wasn't referring to an iterator that can be told to "start over again"

but

to an iterable object that can produce multiple independent iterators of itself, each one good for a single iteration.

That's right.

The language does make a distinction between an *iterable* object that may have only an __iter__ method and an *iterator* that has a next method. This distinction is blurred a bit by the fact that iterators also have an __iter__ method that also makes them appear as one-shot iterables.

Yep. [Part of the reason I want to know whether I've got a one-shot sequence is that inspecting that sequence then becomes an information-destroying operation -- only being able to touch it once changes how you have to handle it] I was thinking one potentially nice way to introspect about multi-pass-ability might be to get an iterator and to see whether it was copyable. Currently even most multi-pass iterators can't be copied with copy.copy(). -Dave

From: "Oren Tirosh" <oren-py-d@hishome.net>

On Tue, Jul 09, 2002 at 04:43:25AM -0400, David Abrahams wrote:

...

Yep. [Part of the reason I want to know whether I've got a one-shot sequence is that inspecting that sequence then becomes an information-destroying operation -- only being able to touch it once changes how you have to handle it]

I was thinking one potentially nice way to introspect about multi-pass-ability might be to get an iterator and to see whether it was copyable. Currently even most multi-pass iterators can't be copied with copy.copy().

I wouldn't call it a one-shot sequence - it's just an iterator. The name iterator is enough to suggest that it is disposable and good for just one pass through the container.

If the object has an __iter__ method but no next it's not an iterator and therefore most likely re-iterable. One notable exception is a file object. File iterators affect the current position of the file.

No kidding, that's the problem I'm talking about. It does me no good to have a criterion for determinining re-iterability which fails for the case I'm most concerned with ;-)

If you think about it you'll see that file objects aren't really containers - they are already iterators. The real container is the file on the disk

There might not be a "real container" -- if it's an input pipe the data disappears as you iterate it. -Dave

On Tue, Jul 09, 2002 at 08:14:38AM -0400, François Pinard wrote:

This is all clear to me, except for one little thing. I wonder why class `ifile' has an `__iter__' method itself. I know it is said to be the "iterator protocol", and I wonder why it has to be.

I don't like it either. In my previous message about the language 'Mamba' in an alternative universe I have an example of an alternative: if object has a tp_iter it is called, otherwise the object must have a tp_next.

My understanding is that `__iter__' returns an iterator all ready to be enquired a number of times through `.next()' calls, and I presume that if any re-initialisation has to take place, it is within `__iter__'. However, as the iterator maintains its own progressive state, I do not see the intent and purpose of the iterator having an `__iter__' method itself. Would it make sense using the iterator `__iter__' as the preferred place where it re-initialises itself?

As far as I can tell this was done so that for could iterate over both iterables and iterators. I just don't see why it has to be done by all iterators instead of in just one place, adding much confusion between iterators and iterables in the process. Oren

pinard@iro.umontreal.ca:

could we have the guarantee that if an iterable returns an iterator through the iterable's __iter__, the iterator's __iter__ method will never be called from looping over the iterable?

[...pause while Greg's brain parses that sentence...] Yes, I believe that's true. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

[François Pinard]

If yes, then, the Library Reference is misleading, at the page:

http://www.python.org/dev/doc/devel/lib/typeiter.html

when it strongly says that any iterator's __iter__ method is "required". I guess this is a possible source of confusion.

This is how Python the language is defined. The current C Python implementation doesn't enforce it, and you may be able to get away with defining an iterator that doesn't supply an __iter__ method in some contexts under the current C implementation. If you do, though, you're breaking the rules, and there's no guarantee your code will continue to work.

The context does not make it clear that the iterator's __iter__ method is *only* required whenever one *also* wants to use an iterator as an iterable.

That's not how the iteration protocol is defined, and isn't how it should be defined either. Requiring *some* method with a reserved name is an aid to introspection, lest it become impossible to distinguish, say, an iterator from an instance of a doubly-linked list node class that just happens to supply methods named .prev() and .next() for an unrelated purpose.

Better would be to describe __iter__ only once, the first time through, saying everything there that has to be said, and only retain for iterators the requirement of having a `next()' method. We should describe the truth.

Except that iterators are required to have an __iter__ method: this is a matter of definition, not just of reverse-engineering the minimum you can get away with under the current implementation in assorted contexts. You'll discover this hard way the first time you try to pass an iterator without an __iter__ method to a routine you didn't write that says it accepts any iterable object as an argument. Such a routine is entitled-- by the documented requirements --to rely on its argument responding sensibly to an __iter__ message.

P.S. - Also, I do not understand the tiny bit about the `in' statement in the above page. Has `in' ever been a statement?

I figure you're talking about this: ... to be used with the for and in statements The tail end of that is indeed worded poorly; 'in' isn't a statement.

If it refers to the comparison operator `in',

Yes, that's the intent.

then has it any special properties when used with iterators?

In x in y y can be any iterable object. As an extreme example, if "error\n" in file('msgs'):

David> Why bother with __multiter__? If you can distinguish a multiple David> iterator by the presence of __copy__, You can always do David> hasattr(x.__iter__(),"__copy__") to find out whether something David> is multi-iteratable. Because explicit is better than implicit :-) More seriously, I can imagine distinguishing a multiple iterator by the presence of __copy__, but I can't imagine using the presence of __copy__ to determine whether a *container* supports multiple iteration. For example, there surely exist containers today that support __copy__ but whose __iter__ methods yield iterators that do not themselves support __copy__. Another reason is that I can imagine this idea extended to encompass, say, ambidextrous iterators that support prev() as well as next(), and I would want to use __ambiter__ as a marker for those rather than having to create an iterator and see if it has prev().

Ping> Just fetch the iterator from the container and look for __copy__ on that. Yes, that's an alternative. However the purpose my suggestion of __multiter__ was not to use it to test for multiple iteration, but to enable a container to be able to yield either a single or a multiple iterator on request. Ping> Or, what if there is no container to begin with, but the iterator is still Ping> copyable? You can't flag that by putting __multiter__ on anything; again Ping> it makes more sense to just provide __copy__ on the iterator. You could flag it by putting __multiter__ on the iterator, just as iterators presently have __iter__. Ping> All that's really necessary here is to document the convention about what Ping> __copy__ is supposed to mean if it's available on an iterator. If we Ping> all agree that __copy__ should preserve an independent copy of the Ping> current state of the iterator, we're all set. Not quite. We also need an agreement that calling __iter__ on a container is not a destructive operation unless you call next() on the iterator that you get back.

...

Another reason is that I can imagine this idea extended to encompass, say, ambidextrous iterators that support prev() as well as next(), and I would want to use __ambiter__ as a marker for those rather than having to create an iterator and see if it has prev().

Ping> I think a proliferation of iterator-fetching methods would be a Ping> messy and unpleasant prospect. After __iter__, __multiter__, Ping> and __ambiter__, what next? __mutableiter__? Ping> __depthfirstiter__? __breadthfirstiter__? A data structure that supports several different kinds of iteration has to provide that support somehow. What's your suggestion?

...

However the purpose my suggestion of __multiter__ was not to use it to test for multiple iteration, but to enable a container to be able to yield either a single or a multiple iterator on request.

David> Why would you want that? Seems like a corner case at best. You're right -- I wasn't thinking clearly. What I meant to say was that I would like a program that expects to be able to use a multiple iterator to be able to say so simply and efficiently in code. For example: for i in multiter(x): // whatever I would like this to fail cleanly if x does not support multiple iterators.

...

A data structure that supports several different kinds of iteration has to provide that support somehow. What's your suggestion?

David> class DataStructure(object): David> def __init__(self): David> self._numbers = range(10); David> self._names = [ str(x) for x in range(10) ]; David> names = property(lambda self: iter(self._names)) David> numbers = property(lambda self: iter(self._numbers)) David> x = DataStructure(); David> for y in x.names: David> print repr(y), David> print David> for y in x.numbers: David> print repr(y), David> [Y'know, Python is great. That worked the first time I ran it.] I don't understand how this code answers my question. You've asked for iterators over two different data structures. What I was asking was, for example, how one might arrange for a single tree to yield either a depth-first or breadth-first iterator.

David> Just replace 'names' by breadth_first and 'numbers' by depth_first. David> or-vice-versa-ly y'rs, which doesn't address the question of a uniform convention.

David> I'm with you on the desire to have a way to get a David> multipass-iterator-or-error in one swell foop. That says "I David> want to iterate over this thing without changing it". I still David> think that hasattr(iter(x), '__copy__') is a pretty clean way David> to do that, despite the fact that it potentially creates an David> iterator (which some people apparently view as too heavyweight David> as an introspection step). In particular, creating an iterator had better not be a destructive operation. David> However, I don't see any point in trying to define a protocol David> for every different possible iteration view of a thing. Dicts David> have keys, values, and items. Trees have breadth-first, David> depth-first, inorder, postorder, blah, blah, blah. There are David> just too many of these, and they're all different. I wasn't suggesting defining a protocol for every possible iteration view. I was raising the question of whether multi-pass iteration was likely to be a common enough operation that it is worth defining a protocol for it, while leaving the door open to defining protocols for others should it turn out to be desirable to do so.

[David Abrahams]

[Y'know, Python is great. That worked the first time I ran it.]

Oops! Please file a bug report on SourceForge <wink>.

An aside on "next" vs. "__next__":

Note that this convention would also suggest that "next" should be called "__next__", since "for" calls "next" implicitly. I forget why we ended up going with "next" instead of "__next__". I think "__next__" would have been better, especially in light of this:

Tim Peters wrote:

...

Requiring *some* method with a reserved name is an aid to introspection, lest it become impossible to distinguish, say, an iterator from an instance of a doubly-linked list node class that just happens to supply methods named .prev() and .next() for an unrelated purpose.

This is exactly why the iterator protocol should consist of one method named "__next__" rather than two methods named "__iter__" (which has nothing to do with the act of iterating!) and "next" (which is the one we really care about, but can collide with existing method names).

As far as i know, "next" is the only implicitly-called method of an internal protocol that has no underscores. It's a little late to fix the name of "next" in Python 2, though it might be worth considering for Python 3.

Yup. I regret this too. We should have had a built-in next(x) which calls x.__next__(). I think that if it had been __next__() we wouldn't have the mistake that I just discovered -- that all the iterator types that define a next() method shouldn't have done so, because you get one automatically which is the tp_iternext slot wrapped. :-( But yes, it's too late to change now. --Guido van Rossum (home page: http://www.python.org/~guido/)

On Tue, Jul 16, 2002 at 09:29:55PM -0400, Guido van Rossum wrote: | > An aside on "next" vs. "__next__": | > | > As far as i know, "next" is the only implicitly-called method of | > an internal protocol that has no underscores. It's a little late | > to fix the name of "next" in Python 2, though it might be worth | > considering for Python 3. | | Yup. I regret this too. | But yes, it's too late to change now. I don't think it is too late. 90% ++ of the python code base out there doesn't use iterators yet... people are still wrapping their minds around it to see how they can use it in their applications. If it was publicly stated that this could be "fixed" in the next version I don't think that it would hurt. These things happen, and sometimes its best to "roll back". Programmers understand this. Best, Clark -- Clark C. Evans Axista, Inc. http://www.axista.com 800.926.5525 XCOLLA Collaborative Project Management Software

From: "Barry A. Warsaw" <barry@zope.com>

CC> I don't think it is too late. 90% ++ of the python code base CC> out there doesn't use iterators yet... people are still CC> wrapping their minds around it to see how they can use it in CC> their applications. If it was publicly stated that this could CC> be "fixed" in the next version I don't think that it would CC> hurt. These things happen, and sometimes its best to "roll CC> back". Programmers understand this.

And besides (to continue Clark's devils advocacy), how much of the code out there that /does/ use iterators, calls .next() explicitly?

Hmm, I'm getting excited! We rarely get an opportunity to fix mistakes in language design. Probably someone will bring me back to reality shortly, though ;-) Maybe I'll do it: the problem is really the iterators people have written. However, you could implicitly generate __next__() which calls next() when the result of __iter__() lacks a __next__() function... with a warning, of course. -Dave

On Wed, Jul 17, 2002 at 10:09:13AM -0400, Guido van Rossum wrote: | > I don't think it is too late. 90% ++ of the python code base out | > there doesn't use iterators yet... people are still wrapping their | > minds around it to see how they can use it in their applications. | > If it was publicly stated that this could be "fixed" in the next | > version I don't think that it would hurt. These things happen, | > and sometimes its best to "roll back". Programmers understand this. | | I find this really hard to believe, given that such a big deal has | been made of iterators. i None of my code uses explicit use of iterators, and I was very aware of them. My new code that I'm building now does, but it wouldn't take much effort to fix it. I myself personally would rather keep Python "clean" of blemish. For the most part, Python is really free of dragons and that's why I like it. I'm willing to put up with short-term pain for long term gain. Unlike Java or Visual Basic, I intend to be programming in Python 10+ years from now; so from my perspective, it is an investment. Plus, most features don't get used by the public for at least a year or so as it takes a while for the code-examples to start using them and books to be updated. | Care to conduct a survey on c.l.py? Sure. I'll run the survey and report back. What would be the options? It'll be a simple CGI form using a radio or check boxes and a button. I'll aggregate the results. To do this I need: - A specific description of what would change - An example of what would break, plus what it would be replaced with. - An explanation of what problems occur when the blemish isn't fixed (what can't you do?) | Given that it's really only a very minor problem, I'd rather not | expend the effort to 'fix" this. Well, if it is a minor problem, it shouldn't be that hard to fix. *evil grins* Clark -- Clark C. Evans Axista, Inc. http://www.axista.com 800.926.5525 XCOLLA Collaborative Project Management Software

On Wed, Jul 17, 2002 at 11:03:48AM -0400, Guido van Rossum wrote: | | Calling it a dragon sounds way overstated. Oh. I wasn't calling it a dragon... I was stating that Python is dragon free. | > | Care to conduct a survey on c.l.py? | > | > Sure. I'll run the survey and report back. Ok. Here is the survey form for comment before it is posted. http://yaml.org/wk/survey?id=pyiter I'll summarize the results after the survey has run its course... Best, Clark http://yaml.org

Guido:

- The mapping between the next() method and the tp_iternext slot in the type object would disappear, and instead the __next__() method would be mapped to this slot.

For what it's worth, I took it upon myself to "fix" this already in Pyrex extension types. So if you make this change, you'll be making Python more compatible with Pyrex. :-) Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

Guido van Rossum wrote:

...

- There really isn't anything "broken" about the current situation; it's just that "next" is the only method name mapped to a slot in the type object that doesn't have leading and trailing double underscores.

Are you saying the _only_ reason to rename it is for consistency with the other type slot method names? That's really weak, IMHO, and not worth any kind of backwards incompatibility (which seems unavoidable).

Neil

Almost. This means that we're retroactively saying that all objects with a next method are iterators, thereby slightly stomping on the user's namespace. But as long a you don't use such an object as an iterator, it's harmless. And if my position wasn't clear already, I agree it's not worth "fixing". :-) --Guido van Rossum (home page: http://www.python.org/~guido/)

[Patrick K. O'Brien]

So perhaps we need some sort of concept of a "grace period" on brand-new features during which blemishes can be polished off, even if the polishing breaks backward compatibility. [...] Would something like this be an acceptable compromise?

I know it was not the original intent of importing from __future__, but maybe this could be linked with __future__ as well. People wanting guaranteed stability should just never import from __future__ at all, It's just an idea, I'm not pushing for it. I do not even like it... For one, I'm quite ready to adjust the things I'm responsible for, whenever the need arises, not being part of the bullet tied to Python development. On the other hand, I know administrators not far from me that get very upset when they learn about specification changes for any software they rely on for production, and I do understand their need for a peaceful life. Surely, it's not easy to please everybody. In French, there is this nice proverb, which is in fact the last verses of one of LaFontaine's fables: "On ne peut, dit le meunier, plaire à tout le monde et à son père: bien faire et laisser braire!". In word, that means "do well and let mumble!". :-) -- François Pinard http://www.iro.umontreal.ca/~pinard

[Oren Tirosh]

On Sun, Aug 04, 2002 at 04:07:08PM -0500, Patrick K. O'Brien wrote:

...

[Guido van Rossum]

...

- There really isn't anything "broken" about the current situation; it's just that "next" is the only method name mapped to a slot in the type object that doesn't have leading and trailing double underscores.

But would you define it as __next__() if you had to do it again? A __next__()/next() relationship does seem to fit more neatly.

...

I'm way behind on the email for this list, but I wanted to chime in with an idea related to this old thread. I know we want to limit the rate of language/feature changes for the business community. At the same time, this situation with iterators is proof that even the best thought out new features can still have a few blemishes that get discovered after they've been incorporated into Python proper.

I think I have a reasonable solution for the re-iteration blemish in the iteration protcol without breaking backward compatibility:

http://mail.python.org/pipermail/python-dev/2002-July/026960.html

...

So perhaps we need some sort of concept of a "grace period" on brand-new features during which blemishes can be polished off, even if the

...

breaks backward compatibility. After the grace period, breaking backward compatibility becomes a higher priority.

Giving more people a chance to play with new features before they are finalized is a very good idea. When a significant new feature is checked in to the CVS a preview version can be released in source and precompiled

polishing form

to encourage more people to test it. Most CVS snapshots seem stable enough for a programmer's daily use.

Given the general lack of alpha- and beta-testing there'd be very little feedback. I seem to remember that the CVS snapshots went missing in action recently without anyone noticing, which shows that they aren't much used, and I guess the same would be true of preview versions. Tracking the CVS repository will test such features, but getting more testing than that would be difficult. I *do* agree that such feature testing would be inestimably useful.

A good example of such a significant new feature is the source encoding just checked in.

Indeed. regards ----------------------------------------------------------------------- Steve Holden http://www.holdenweb.com/ Python Web Programming http://pydish.holdenweb.com/pwp/ -----------------------------------------------------------------------

On Wed, Jul 17, 2002, Clark C . Evans wrote:

Sure. I'll run the survey and report back. What would be the options? It'll be a simple CGI form using a radio or check boxes and a button. I'll aggregate the results.

Make sure you ask for e-mail addresses to prevent duplicates. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Project Vote Smart: http://www.vote-smart.org/

But i think this is more than a minor problem. This is a namespace collision problem, and that's significant. Naming the method "next" means that any object with a "next" method cannot be adapted to support the iterator protocol. Unfortunately "next" is a pretty common word and it's quite possible that such a method name is already in use.

Can you explain this? Last time I checked CVS, PEP 246 wasn't implemented yet, so I don't think you mean "adapted" in that sense. Generally speaking, iterator implementations aren't created by making changes to an existing class -- they're created by creating new a class. The only change to *existing* classes needed is the addition of an __iter__ method to the underlying container object. So I'm not sure what you mean. --Guido van Rossum (home page: http://www.python.org/~guido/)

Well, i guess that's part of the protocol philosophy. There exist cursor-like objects that would be natural candidates for being used like iterators (files are one example, database cursors are another). Choosing __next__ makes it possible to add support to an existing object when appropriate, instead of requiring an auxiliary object regardless of whether it's appropriate or inappropriate.

OK, that's clear.

To me, the former feels like the more typical Python thing to do, because it's consistent with the way all the other protocols work. So it's from this perspective that "next" without underscores is a wart to me.

Yes.

For example, when something is container-like you can implement __getitem__ on the object itself, and then you can use [] with the object. Some objects let you fetch containers and some objects implement __getitem__ on their own. But we don't force everybody to provide a convert-to-container operation in all cases before allowing them to provide __getitem__.

Correct. Now, weren't you a co-author of the Iterator PEP? I wish you'd brought this up then. Or maybe you did, and I overruled you. Sorry then. But I don't think we can withdraw this so easily. It's not the end of the world. --Guido van Rossum (home page: http://www.python.org/~guido/)

I do think there is some potential for errors caused by misunderstandings about whether or not "for x in y" is destructive. That's the thing that worries me the most. I think this is the main reason why the old practice of abusing __getitem__ was bad, and thus helped to motivate iterators in the first place. It seems serious enough that migrating to something that distinguishes destructive-for from non-destructive-for could indeed be worth the cost.

I'm not sure I understand this (this seems to be my week for not understanding what people write :-( ). First of all, I'm not sure what exactly the issue is with destructive for-loops. If I have a function that contains a for-loop over its argument, and I pass iter(x) as the argument, then the iterator is destroyed in the process, but x may or may not be, depending on what it is. Maybe the for-loop is a red herring? Calling next() on an iterator may or may not be destructive on the underlying "sequence" -- if it is a generator, for example, I would call it destructive. Perhaps you're trying to assign properties to the iterator abstraction that aren't really there? Next, I'm not sure how renaming next() to __next__() would affect the situation w.r.t. the destructivity of for-loops. Or were you talking about some other migration? --Guido van Rossum (home page: http://www.python.org/~guido/)

On Fri, Jul 19, 2002, Ka-Ping Yee wrote:

I believe this is where the biggest debate lies: whether "for" should be non-destructive. I realize we are currently on the other side of the fence, but i foresee enough potential pain that i would like you to consider the value of keeping "for" loops non-destructive.

Consider for line in f.readlines(): in any version of Python. Adding iterators made this more convenient and efficient, but I just can't see your POV in the general case. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Project Vote Smart: http://www.vote-smart.org/

aahz wrote:

...

I believe this is where the biggest debate lies: whether "for" should be non-destructive. I realize we are currently on the other side of the fence, but i foresee enough potential pain that i would like you to consider the value of keeping "for" loops non-destructive.

Consider

for line in f.readlines():

in any version of Python.

and? for-in doesn't modify the object returned by f.readlines(), and never has. </F>

aahz wrote:

While technically true, that seems to be sidestepping the point from my POV.

really? are you arguing that when Ping says that for-in shouldn't destroy the target, he's really saying that python shouldn't allow methods to have side effects if they can be called from an expression used in a for-in statement? why would he say that?

I think that few people see for loops as inherently non-destructive due to the use case I presented above.

I think most people can tell the difference between an object and a method with side-effects. I doubt they would be able to get much done in Python if they couldn't.

Beyond that, the for loop is itself inherently mutating in Python older than 2.2

in what sense? it calls the object's __getitem__ method with an integer index value, until it gets an IndexError. in what way is that "inherently mutating"? </F>

...

...

...

...

"A" == Aahz <aahz@pythoncraft.com> writes:

A> The for loop in Python <2.2 has an internal state object. A> Iterators are the external manifestation of that state object, A> generalized to objects other than sequences. I'm surprised A> that anyone is surprised that the state object gets A> mutated/destroyed. I'm also surprised that people are A> surprised about what happens when that state object is coupled A> to an inherently mutating object such as file objects. Well said. -Barry

On Fri, Jul 19, 2002, Paul Svensson wrote:

Pending the pain of the yet unseen migration plan, I'm +1 on removing __iter__ from all core iterators +1 on renaming next() to __next__() +1 on presenting file objects as iterators rather than iterables +0 on the new 'for x from y' syntax

I'd vote this way: -0 on removing __iter__ +1 on renaming next() to __next__() +0 on presenting file objects as iterators +1 on finishing up the patch that fixes the xreadlines() mess -1 on for x from y -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Project Vote Smart: http://www.vote-smart.org/

[Ping]

... I believe this is where the biggest debate lies: whether "for" should be non-destructive. I realize we are currently on the other side of the fence, but i foresee enough potential pain that i would like you to consider the value of keeping "for" loops non-destructive.

I'm having a hard time getting excited about this. If you had made this argument before the iterator protocol was implemented, it may have been more or less intriguing. But it was implemented and released some time ago, and I just haven't seen any evidence of such problems on c.l.py, the Help list, or the Tutor list (all of which I still pay significant attention to). "for" did and does work in accord with a simple protocol, and whether that's "destructive" depends on how the specific objects involved implement their pieces of the protocol, not on the protocol itself. The same is true of all of Python's hookable protocols. What's so special about "for" that it should pretend to deliver purely functional behavior in a highly non-functional language? State mutates. That's its purpose <wink>.

On Fri, 19 Jul 2002, Tim Peters wrote:

"for" did and does work in accord with a simple protocol, and whether that's "destructive" depends on how the specific objects involved implement their pieces of the protocol, not on the protocol itself. The same is true of all of Python's hookable protocols.

Name any protocol for which the question "does this mutate?" has no answer. (I ask you to accept that __call__ is a special case.)

What's so special about "for" that it should pretend to deliver purely functional behavior in a highly non-functional language?

Who said anything about functional behaviour? I'm not requiring that looping *never* mutate. I just want to be able to tell *whether* it will. -- ?!ng

On Sun, 21 Jul 2002, Tim Peters wrote:

x1 = y.z x2 = y.z

Are x1 and x2 the same object after that? At least equal? Did either line mutate y? You simply can't know without knowing how y's type implements __getattr__, and with the introduction of computed attributes (properties) it's just going to get muddier.

That's not the point. You could claim that *any* polymorphism in Python is useless by the same argument. But Python is not useless; Python code really is reusable; and that's because there are good conventions about what the behaviour *should* be. People who do really find this upsetting should go use a strongly-typed language. In general, getting "y.z" should be idempotent, and should not mutate y. I think everyone would agree on the concept. If it does mutate y with visible effects, then the implementor is breaking the convention. Sure, Python won't prevent you from writing a file-like class where you write the string "blah" to the file by fetching f.blah and you close the file by mentioning f[42]. But when users of this class then come running after you with pointed sticks, i'm not going to fight them off. :) This is a list of all the type slots accessible from Python, before iterators (i.e. pre-2.2). Beside each is the answer to the question: Suppose you look at the value of x, then do this operation to x, then look at the value of x. Should we expect the two observed values to be the same or different? nb_add same nb_subtract same nb_multiply same nb_divide same nb_remainder same nb_divmod same nb_power same nb_negative same nb_positive same nb_absolute same nb_nonzero same nb_invert same nb_lshift same nb_rshift same nb_and same nb_xor same nb_or same nb_coerce same nb_int same nb_long same nb_float same nb_oct same nb_hex same nb_inplace_add different nb_inplace_subtract different nb_inplace_multiply different nb_inplace_divide different nb_inplace_remainder different nb_inplace_power different nb_inplace_lshift different nb_inplace_rshift different nb_inplace_and different nb_inplace_xor different nb_inplace_or different nb_floor_divide same nb_true_divide same nb_inplace_floor_divide different nb_inplace_true_divide different sq_length same sq_concat same sq_repeat same sq_item same sq_slice same sq_ass_item different sq_ass_slice different sq_contains same sq_inplace_concat different sq_inplace_repeat different mp_length same mp_subscript same mp_ass_subscript different bf_getreadbuffer same bf_getwritebuffer same bf_getsegcount same bf_getcharbuffer same tp_print same tp_getattr same tp_setattr different tp_compare same tp_repr same tp_hash same tp_call ? tp_str same tp_getattro same tp_setattro different In every case except for __call__, there exists a canonical answer. We all rely on these conventions every time we write a Python program. And learning these conventions is a necessary part of learning Python. You can argue, as Guido has, that in the particular case of for-loops distinguishing between mutating and non-mutating behaviour is not worth the trouble. But you can't say that we should give up on the whole concept *in general*. -- ?!ng

It's just not the way i expect for-loops to work. Perhaps we would need to survey people for objective data, but i feel that most people would be surprised if

for x in y: print x for x in y: print x

did not print the same thing twice, or if

if x in y: print 'got it' if x in y: print 'got it'

did not do the same thing twice. I realize this is my own opinion, but it's a fairly strong impression i have.

I think it's a naive persuasion that doesn't hold under scrutiny. For a long time people have faked iterators by providing pseudo-sequences that did unspeakable things. In general, I'm pretty sure that if I asked an uninitiated user what "for line in file" would do, if it did anything, they would understand that if you tried that a second time you'd hit EOF right away.

Even if it's okay for for-loops to destroy their arguments, i still think it sets up a bad situation: we may end up with functions manipulating sequence-like things all over, but it becomes unclear whether they destroy their arguments or not. It becomes possible to write a function which sometimes destroys its argument and sometimes doesn't. Bugs get deeper and harder to find.

This sounds awfully similar to the old argument "functions (as opposed to procedures) should never have side effects". ABC implemented that literally (the environment was saved and restored around function calls, with an exception for the seed for the built-in random generator), with the hope that it would provide fewer surprises. It did the opposite: it drove people crazy because the language was trying to be smarter than them.

I believe this is where the biggest debate lies: whether "for" should be non-destructive. I realize we are currently on the other side of the fence, but i foresee enough potential pain that i would like you to consider the value of keeping "for" loops non-destructive.

I don't see any real debate. I only see you chasing windmills. Sorry. For-loops have had the possibility to destroy their arguments since the day __getitem__ was introduced.

...

Maybe the for-loop is a red herring? Calling next() on an iterator may or may not be destructive on the underlying "sequence" -- if it is a generator, for example, I would call it destructive.

Well, for a generator, there is no underlying sequence.

while 1: print next(gen)

makes it clear that there is no sequence, but

for x in gen: print x

seems to give me the impression that there is.

This seems to be a misrepresentation. The idiom for using any iterator (not just generators) *without* using a for-loop would have to be something like: while 1: try: item = it.next() # or it.__next__() or next(it) except StopIteration: break ...do something with item... (Similar to the traditional idiom for looping over the lines of a file.) The for-loop over an iterator was invented so you could write this as: for item in it: ...do something with item... I'm not giving that up so easily!

...

Perhaps you're trying to assign properties to the iterator abstraction that aren't really there?

I'm assigning properties to "for" that you aren't. I think they are useful properties, though, and worth considering.

I'm trying to be open-minded, but I just don't see it. The for loop is more flexible than you seem to want it to be. Alas, it's been like this for years, and I don't think the for-loop needs a face lift.

I don't think i'm assigning properties to the iterator abstraction; i expect iterators to destroy themselves. But the introduction of iterators, in the way they are now, breaks this property of "for" loops that i think used to hold almost all the time in Python, and that i think holds all the time in almost all other languages.

Again, the widespread faking of iterators using destructive __getitem__ methods that were designed to be only used in a for-loop defeats your assertion.

...

Next, I'm not sure how renaming next() to __next__() would affect the situation w.r.t. the destructivity of for-loops. Or were you talking about some other migration?

The connection is indirect. The renaming is related to: (a) making __next__() a real, honest-to-goodness protocol independent of __iter__;

next() is a real, honest-to-goodness protocol now, and it is independent of __iter__() now.

and (b) getting rid of __iter__ on iterators. It's the presence of __iter__ on iterators that breaks the non-destructive-for property.

So you prefer the while-loop version above over the for-loop version? Gotta be kidding.

I think the renaming of next() to __next__() is a good idea in any case. It is distant enough from the other issues that it can be done independently of any decisions about __iter__.

Yeah, it's just a pain that it's been deployed in Python 2.2 since last December, and by the time 2.3 is out it will probably have been at least a full year. Worse, 2.2 is voted to be Python-in-a-Tie, giving that particular idiom a very long lifetime. I simply don't think we can break compatibility that easily. Remember the endless threads we've had about the pace of change and stability. We have to live with warts, alas. And this is a pretty minor one if you ask me. (I realize that you're proposing another way out in a separate message. I'll reply to that next. Since you changed the subject, I can't wery well reply to it here.) --Guido van Rossum (home page: http://www.python.org/~guido/)

Aahz wrote:

On Fri, Jul 19, 2002, Guido van Rossum wrote:

...

Ping:

...

I think the renaming of next() to __next__() is a good idea in any case. It is distant enough from the other issues that it can be done independently of any decisions about __iter__.

Yeah, it's just a pain that it's been deployed in Python 2.2 since last December, and by the time 2.3 is out it will probably have been at least a full year. Worse, 2.2 is voted to be Python-in-a-Tie, giving that particular idiom a very long lifetime. I simply don't think we can break compatibility that easily. Remember the endless threads we've had about the pace of change and stability. We have to live with warts, alas. And this is a pretty minor one if you ask me.

Is this a Pronouncement, or are we still waiting on the results of the survey? Note that several people have suggested a multi-release strategy for fixing this problem; does that make any difference?

Would it be good to use __next__() if it exists, else try next()? This doesn't fix the current 'wart,' however, it could allow moving closer to the desired end. It could cause confusion. For compatability, one would only need to do: next = __next__ or vica versa. Not sure this is worth it. But if there is a transition, it could ease the pain. Neal

...

Yeah, it's just a pain that it's been deployed in Python 2.2 since last December, and by the time 2.3 is out it will probably have been at least a full year. Worse, 2.2 is voted to be Python-in-a-Tie, giving that particular idiom a very long lifetime. I simply don't think we can break compatibility that easily. Remember the endless threads we've had about the pace of change and stability. We have to live with warts, alas. And this is a pretty minor one if you ask me.

Is this a Pronouncement, or are we still waiting on the results of the survey?

That is my current opinion. I'm waiting for the results of the survey to see if I'll be swayed (but I don't think it's likely).

Note that several people have suggested a multi-release strategy for fixing this problem; does that make any difference?

Such a big gun for such a minor problem. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido:

Generally speaking, iterator implementations aren't created by making changes to an existing class

Continuing with my scanner example, it's conceivable that I might want to give it an iterator interface as an alternative to the existing one -- it's already an iterator, really, it just doesn't have the new standard iterator interface. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

On Wed, 17 Jul 2002, Clark C . Evans wrote:

On Wed, Jul 17, 2002 at 02:58:55PM -0500, Ka-Ping Yee wrote: | Naming | the method "next" means that any object with a "next" method | cannot be adapted to support the iterator protocol.

Right, but such objects wouldn't be mis-leading beacuse they'd be missing a __iter__ method, correct?

__iter__ is a red herring. It has nothing to do with the act of iterating. It exists only to support the use of "for" directly on the iterator. Iterators that currently implement "next" but not "__iter__" will work in some places and not others. For example, given this: class Counter: def __init__(self, last): self.i = 0 self.last = last def next(self): self.i += 1 if self.i > self.last: raise StopIteration return self.i class Container: def __init__(self, size): self.size = size def __iter__(self): return Counter(self.size) This will work: >>> for x in Container(3): print x ... 1 2 3 But this will fail: >>> for x in Counter(3): print x ... Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: iteration over non-sequence It's more accurate to say that there are two distinct protocols here. 1. An object is "for-able" if it implements __iter__ or __getitem__. This is a subset of the sequence protocol. 2. An object can be iterated if it implements next. The Container supports only protocol 1, and the Counter supports only protocol 2, with the above results. Iterators are currently asked to support both protocols. The semantics of iteration come only from protocol 2; protocol 1 is an effort to make iterators look sorta like sequences. But the analogy is very weak -- these are "sequences" that destroy themselves while you look at them -- not like any typical sequence i've ever seen! The short of it is that whenever any Python programmer says "for x in y", he or she had better be darned sure of whether this is going to destroy y. Whatever we can do to make this clear would be a good idea. -- ?!ng

I wrote:

__iter__ is a red herring. [...blah blah blah...] The short of it is that whenever any Python programmer says "for x in y", he or she had better be darned sure of whether this is going to destroy y. Whatever we can do to make this clear would be a good idea.

Guido wrote:

This is a very good summary of the two iterator protocols. Ping, would you mind adding this to PEP 234?

And i thought it was a critique. Fascinating, Captain. :) I'm happy to add the text, but i want to be clear, then: is it acceptable to write an iterator that only provides <next> if you only care about the "iteration protocol" and not the "for-able protocol"? I see that "ought to" is the most opinion the PEP is willing to give on the topic: A class is a valid iterator object when it defines a next() method that behaves as described above. A class that wants to be an iterator also ought to implement __iter__() returning itself. -- ?!ng

...

This is a very good summary of the two iterator protocols. Ping, would you mind adding this to PEP 234?

And i thought it was a critique. Fascinating, Captain. :)

I'm happy to add the text, but i want to be clear, then: is it acceptable to write an iterator that only provides <next> if you only care about the "iteration protocol" and not the "for-able protocol"?

No, an iterator ought to provide both, but it's good to recognize that there *are* two protocols.

I see that "ought to" is the most opinion the PEP is willing to give on the topic:

A class is a valid iterator object when it defines a next() method that behaves as described above. A class that wants to be an iterator also ought to implement __iter__() returning itself.

I would like to see this strengthened. I envision "iterator algebra" code that really needs to be able to do a for loop over an iterator when it feels like it. --Guido van Rossum (home page: http://www.python.org/~guido/)

Maybe the reasons behind having __iter__() returning itself should be clearly expressed in the PEP, too. On this list, Tim gave one recently, Guido gives another here, but unless I missed it, the PEP gives none. Usually, PEPs explain the reasons behind the choices.

Ping added this to the PEP: The two methods correspond to two distinct protocols: 1. An object can be iterated over with "for" if it implements __iter__() or __getitem__(). 2. An object can function as an iterator if it implements next(). Container-like objects usually support protocol 1. Iterators are currently required to support both protocols. The semantics of iteration come only from protocol 2; protocol 1 is present to make iterators behave like sequences. But the analogy is weak -- unlike ordinary sequences, iterators are "sequences" that are destroyed by the act of looking at their elements. (I could do without the last sentence, since this expresses a value judgement rather than fact -- not a good thing to have in a PEP's "specification" section.) --Guido van Rossum (home page: http://www.python.org/~guido/)

What about:

"...sequences. Note that the act of looking at an iterator's elements mutates the iterator."

That doesn't belong in the spec either -- nothing requires an iterator to have mutable state, let alone to mutate it when next() is called.

I wrote:

__iter__ is a red herring. [...blah blah blah...] Iterators are currently asked to support both protocols. The semantics of iteration come only from protocol 2; protocol 1 is an effort to make iterators look sorta like sequences. But the analogy is very weak -- these are "sequences" that destroy themselves while you look at them -- not like any typical sequence i've ever seen!

The short of it is that whenever any Python programmer says "for x in y", he or she had better be darned sure of whether this is going to destroy y. Whatever we can do to make this clear would be a good idea.

On Wed, 17 Jul 2002, Guido van Rossum wrote:

This is a very good summary of the two iterator protocols. Ping, would you mind adding this to PEP 234?

I have now done so. I didn't add the whole thing verbatim, because the tone doesn't fit: it was written with the intent of motivating a change to the protocol, rather than describing what the protocol is. Presumably we don't want the PEP to say "__iter__ is a red herring". There's a bunch of issues flying around here, which i'll try to explain better in a separate posting. But i wanted to take care of Guido's request first. I have toned down and abridged my text somewhat, and strengthened the requirement for __iter__(). Here is what the "API specification" section now says: Classes can define how they are iterated over by defining an __iter__() method; this should take no additional arguments and return a valid iterator object. A class that wants to be an iterator should implement two methods: a next() method that behaves as described above, and an __iter__() method that returns self. The two methods correspond to two distinct protocols: 1. An object can be iterated over with "for" if it implements __iter__() or __getitem__(). 2. An object can function as an iterator if it implements next(). Container-like objects usually support protocol 1. Iterators are currently required to support both protocols. The semantics of iteration come only from protocol 2; protocol 1 is present to make iterators behave like sequences. But the analogy is weak -- unlike ordinary sequences, iterators are "sequences" that are destroyed by the act of looking at their elements. Consequently, whenever any Python programmer says "for x in y", he or she must be sure of whether this is going to destroy y. -- ?!ng

On Wed, 17 Jul 2002, Clark C . Evans wrote:

Right, but such objects wouldn't be mis-leading beacuse they'd be missing a __iter__ method, correct?

Oh, i guess i didn't properly answer your question. Oops. :) My answer would be: you could say that, but wouldn't it suck to have to check for the existence of __iter__ every time you wanted to call next? You can legislate that everyone should implement __iter__ together with next; you can legislate that everyone should check for __iter__ before calling next. To some extent you have to do both or neither; one without the other is inconsistent and would lead to surprises. In practice no one's going to check. So in practice __iter__ isn't really part of the protocol. -- ?!ng

On Wed, Jul 17, 2002 at 02:58:55PM -0500, Ka-Ping Yee wrote: | But i think this is more than a minor problem. This is a | namespace collision problem, and that's significant. Naming | the method "next" means that any object with a "next" method | cannot be adapted to support the iterator protocol. Unfortunately | "next" is a pretty common word and it's quite possible that such | a method name is already in use. Ping, Do you have any suggestions for re-wording the Iterator questionare at http://yaml.org/wk/survey?id=pyiter to reflect this paragraph above? Best, Clark -- Clark C. Evans Axista, Inc. http://www.axista.com 800.926.5525 XCOLLA Collaborative Project Management Software

Ping> On Mon, 15 Jul 2002, Andrew Koenig wrote:

...

However the purpose my suggestion of __multiter__ was not to use it to test for multiple iteration, but to enable a container to be able to yield either a single or a multiple iterator on request.

Ping> I see what you want, though i have a hard time imagining a situation Ping> where it's really necessary to have both (as opposed to just the Ping> multiple iterator, which is strictly more capable). I can certainly Ping> see how you might want to be able to ask for a breadth-first or Ping> depth-first iterator on a tree, though. How about a class that represents a file? If you ask it for a single iterator, that's easy. If you ask it for a multiple iterator, it checks whether the file is really an interactive device such as a pipe or a keyboard. If so, it uses a buffering mechanism to simulate multiple iteration; otherwise, it lets the multiple iterators access the file directly. Then when you ask to iterate over the file, you automatically get the least cumbersome mechanism needed to support the kind of iteration that you requested.

...

...

Or, what if there is no container to begin with, but the iterator is still copyable? You can't flag that by putting __multiter__ on anything; again it makes more sense to just provide __copy__ on the iterator.

...

You could flag it by putting __multiter__ on the iterator, just as iterators presently have __iter__.

Ping> Ugh. I don't like this, for the reasons i outlined in another message: Ping> an iterator is not the same as a container. Iterators always mutate; Ping> containers usually do not (at least not as a result of looking at the Ping> elements). The scenario is this: def search(thing): iter = thing.__multiter__() // now either iter is an iterator that supports __copy__ // or we will raise an exception (and raise it here, rather // than waiting for the first time we try to copy iter).

...

Not quite. We also need an agreement that calling __iter__ on a container is not a destructive operation unless you call next() on the iterator that you get back.

Ping> What i'd like is an agreement that calling __iter__ on a container is Ping> not a destructive operation at all. If it's destructive, then what you Ping> started with is not really a container, and we should encourage people Ping> to call attention to this irregularity in their documentation. Is a file a container or not? Isn't making an iterator from a file and calling next() on it a destructive operation?

...

...

I think a proliferation of iterator-fetching methods would be a messy and unpleasant prospect. After __iter__, __multiter__, and __ambiter__, what next? __mutableiter__? __depthfirstiter__? __breadthfirstiter__?

...

A data structure that supports several different kinds of iteration has to provide that support somehow.

Ping> Agreed. I was unclear: what makes me uncomfortable is the pollution Ping> of the double-underscore namespace. When you do have a container-like Ping> object that supports various kinds of iteration, naturally you are Ping> going to need some methods for getting iterators. I just think it's Ping> not appropriate to establish special names for them. Fair enough. But then why is __iter__ special? Ping> To me, the presence of double-underscores around a method name means Ping> that the method is called automatically. My expectation is that when Ping> i write a method with a "normal" name, the name itself will appear Ping> after a dot wherever that method is used; and that when there's a Ping> method with a "__special__" name, the method is called implicitly. Ping> The implicit call can occur via an operator (e.g. __add__), or to Ping> implement a protocol defined in the language (e.g. __init__), etc. Ping> If you see the string ".__" it means that something unusual is going on. Ping> If you follow this convention, then "__iter__" deserves a special name, Ping> because it is the specially blessed iterator-getter used by "for". Ping> There may be other iterator-getters, but they must be called explicitly, Ping> so they shouldn't get underscores. Ah, is it only "for" that makes __iter__ special, and not iter() ? Ping> An aside on "next" vs. "__next__": Ping> This is exactly why the iterator protocol should consist of one Ping> method named "__next__" rather than two methods named "__iter__" Ping> (which has nothing to do with the act of iterating!) and "next" Ping> (which is the one we really care about, but can collide with Ping> existing method names). Ping> As far as i know, "next" is the only implicitly-called method of Ping> an internal protocol that has no underscores. It's a little late Ping> to fix the name of "next" in Python 2, though it might be worth Ping> considering for Python 3. One way to clarify a discussion of a protocol is to append an "s" and think of a plurality of protocols, so as to see which properites are truly intrinsic and which can vary between protocols. That's part of what I'm trying to do in this discussion. (and I don't presently have a strong opinion about what the right answer is. I don't even know for sure what the question is.)

Gordon> OK, it's a pipe, and one iterator wants to go past Gordon> what's been received. Is that iterator at EOF? Not Gordon> really, just "temporary EOF". So should it block? Gordon> But I'm single threaded and receiving asynchronously. Gordon> Oh, and it turns out to be a humongous download, Gordon> and what happens if the buffering mechanism runs Gordon> out of memory / disk space. Does my process die? Gordon> Aargh. Too much magic. Too many corner cases. The implementation of the file should have a choice: Either refuse to yield a multiple iterator (which seems to be your preference) or yield one that works (which might or might not be my preference, depending on circumstances). In the latter case, I don't think your questions are hard to answer, because most of the answers fall out of the single-iterator case. So if the iterator is at EOF, it should either block or not, depending on what a single iterator should so. The only real question is what happens if the buffering mechanism runs out of space, but that's always a question for such mechanisms; I don't see why it's any more irksome in this particular context.

On Thursday 18 July 2002 01:25 am, Greg Ewing wrote:

Andrew Koenig <ark@research.att.com>:

...

Is a file a container or not?

I would say no, a file object is not a container in Python terms. You can't index it with [] or use len() on it or any of the other things you expect to be able to do on containers.

I think we just have to live with the idea that there are things other than containers that can supply iterators.

Yes, there are such things, and there may be cases in which no other alternative makes sense. But I don't think files are necessarily in such a bind.

Forcing everything that can supply an iterator to bend over backwards to try to be a random-access container as well would be too cumbersome.

Absolutely. But what Oren's patch does, and my mods of it preserve, is definitely NOT "forcing" files "to be random- access containers": on the contrary, it accepts the fact that files aren't containers and conceptually simplifies things by making them iterators instead. I'm not sure about "random access" being needed to be a container. Consider sets, e.g. as per Greg Wilson's soapbox implementation (as modified by my patch to allow immutable-sets, maybe, but that's secondary). They're doubtlessly containers, able to produce on request as many iterators as you wish, each iterator not affecting the set's state in any way -- the ideal. But what sense would it make to force sets to expose a __getitem__? Right now they inherit from dict and thus do happen to expose it, but that's really an implementation artefact showing through (and a good example of why one might like to inherit without needing to expose all of the superclass's interface, to tie this in to another recent thread -- inheritance for implementation). Ideally, sets would expose __contains__, __iter__, __len__, ways to add and remove elements, and perhaps (it's so in Greg's implementation, and I didn't touch that) set ops such as union, intersection &c. someset[anindex] is really a weird thing to have... yet sets _are_ containers! Alex

On Thursday 18 July 2002 20:45, Guido van Rossum wrote: ...

I believe I recommended to Greg to make sets "have" a dict instead of "being" dicts, and I think he agreed. But I guess he never got to implementing that change.

Right. OK, guess I'll make a new patch using delegation instead of inheritance, then. Alex

On Thursday 18 July 2002 09:50 pm, Guido van Rossum wrote:

...

...

I believe I recommended to Greg to make sets "have" a dict instead of "being" dicts, and I think he agreed. But I guess he never got to implementing that change.

Right. OK, guess I'll make a new patch using delegation instead of inheritance, then.

Maybe benchmark the performance too. If the "has" version is much slower, perhaps we could remove unwanted interfaces from the public API by overriding them with something that raises an exception (and rename the internal versions to some internal name if they are needed).

I've just updated patch 580995 with the has-A rather than is-A version. OK, I'll now run some simple benchmarks... Looks good, offhand. Here's the simple benchmark script: import time import set import sys clock = time.clock raw = range(10000) times = [None]*20 print "Timing Set %s (Python %s)" % (set.__version__, sys.version) print "Make 20 10k-items sets (no reps)...", start = clock() for i in times: s10k = set.Set(raw) stend = clock() print stend-start witre = range(1000)*10 print "Make 20 1k-items sets (x10 reps)...", for i in times: s1k1 = set.Set(witre) stend = clock() print stend-start raw1 = range(500, 1500) print "Make 20 more 1k-items sets (no reps)...", for i in times: s1k2 = set.Set(raw1) stend = clock() print stend-start print "20 unions of 1k-items sets 50% overlap...", for i in times: result = s1k1 | s1k2 stend = clock() print stend-start print "20 inters of 1k-items sets 50% overlap...", for i in times: result = s1k1 & s1k2 stend = clock() print stend-start print "20 diffes of 1k-items sets 50% overlap...", for i in times: result = s1k1 - s1k2 stend = clock() print stend-start print "20 simdif of 1k-items sets 50% overlap...", for i in times: result = s1k1 ^ s1k2 stend = clock() print stend-start And here's a few runs (with -O of course) on my PC: [alex@lancelot has]$ python -O ../bench_set.py Timing Set $Revision: 1.5 $ (Python 2.2.1 (#2, Jul 15 2002, 17:32:26) [GCC 2.96 20000731 (Mandrake Linux 8.1 2.96-0.62mdk)]) Make 20 10k-items sets (no reps)... 0.21 Make 20 1k-items sets (x10 reps)... 0.36 Make 20 more 1k-items sets (no reps)... 0.38 20 unions of 1k-items sets 50% overlap... 0.43 20 inters of 1k-items sets 50% overlap... 0.92 20 diffes of 1k-items sets 50% overlap... 1.41 20 simdif of 1k-items sets 50% overlap... 2.38 [alex@lancelot has]$ python -O ../bench_set.py Timing Set $Revision: 1.5 $ (Python 2.2.1 (#2, Jul 15 2002, 17:32:26) [GCC 2.96 20000731 (Mandrake Linux 8.1 2.96-0.62mdk)]) Make 20 10k-items sets (no reps)... 0.22 Make 20 1k-items sets (x10 reps)... 0.37 Make 20 more 1k-items sets (no reps)... 0.39 20 unions of 1k-items sets 50% overlap... 0.44 20 inters of 1k-items sets 50% overlap... 0.93 20 diffes of 1k-items sets 50% overlap... 1.42 20 simdif of 1k-items sets 50% overlap... 2.39 [alex@lancelot has]$ cd ../is [alex@lancelot is]$ python -O ../bench_set.py Timing Set $Revision: 1.5 $ (Python 2.2.1 (#2, Jul 15 2002, 17:32:26) [GCC 2.96 20000731 (Mandrake Linux 8.1 2.96-0.62mdk)]) Make 20 10k-items sets (no reps)... 0.21 Make 20 1k-items sets (x10 reps)... 0.37 Make 20 more 1k-items sets (no reps)... 0.39 20 unions of 1k-items sets 50% overlap... 0.44 20 inters of 1k-items sets 50% overlap... 0.93 20 diffes of 1k-items sets 50% overlap... 1.42 20 simdif of 1k-items sets 50% overlap... 2.38 [alex@lancelot is]$ python -O ../bench_set.py Timing Set $Revision: 1.5 $ (Python 2.2.1 (#2, Jul 15 2002, 17:32:26) [GCC 2.96 20000731 (Mandrake Linux 8.1 2.96-0.62mdk)]) Make 20 10k-items sets (no reps)... 0.22 Make 20 1k-items sets (x10 reps)... 0.38 Make 20 more 1k-items sets (no reps)... 0.4 20 unions of 1k-items sets 50% overlap... 0.44 20 inters of 1k-items sets 50% overlap... 0.93 20 diffes of 1k-items sets 50% overlap... 1.42 20 simdif of 1k-items sets 50% overlap... 2.41 [alex@lancelot is]$ They look much of a muchness to me. Sorry about the version stuck at 1.5 -- forgot to update that, but you can tell the difference by the directory name, 'is' and 'has' resp.:-). Python 2.3 (built from CVS 22 hours ago) is substantially faster at some tasks (intersections and differences): [alex@lancelot has]$ python -O ../bench_set.py Timing Set $Revision: 1.5 $ (Python 2.3a0 (#44, Jul 18 2002, 00:03:05) [GCC 2.96 20000731 (Mandrake Linux 8.2 2.96-0.76mdk)]) Make 20 10k-items sets (no reps)... 0.21 Make 20 1k-items sets (x10 reps)... 0.36 Make 20 more 1k-items sets (no reps)... 0.37 20 unions of 1k-items sets 50% overlap... 0.42 20 inters of 1k-items sets 50% overlap... 0.75 20 diffes of 1k-items sets 50% overlap... 1.08 20 simdif of 1k-items sets 50% overlap... 1.73 [alex@lancelot has]$ python -O ../bench_set.py Timing Set $Revision: 1.5 $ (Python 2.3a0 (#44, Jul 18 2002, 00:03:05) [GCC 2.96 20000731 (Mandrake Linux 8.2 2.96-0.76mdk)]) Make 20 10k-items sets (no reps)... 0.21 Make 20 1k-items sets (x10 reps)... 0.36 Make 20 more 1k-items sets (no reps)... 0.37 20 unions of 1k-items sets 50% overlap... 0.42 20 inters of 1k-items sets 50% overlap... 0.75 20 diffes of 1k-items sets 50% overlap... 1.08 20 simdif of 1k-items sets 50% overlap... 1.74 [alex@lancelot has]$ [alex@lancelot is]$ python -O ../bench_set.py Timing Set $Revision: 1.5 $ (Python 2.3a0 (#44, Jul 18 2002, 00:03:05) [GCC 2.96 20000731 (Mandrake Linux 8.2 2.96-0.76mdk)]) Make 20 10k-items sets (no reps)... 0.21 Make 20 1k-items sets (x10 reps)... 0.35 Make 20 more 1k-items sets (no reps)... 0.37 20 unions of 1k-items sets 50% overlap... 0.41 20 inters of 1k-items sets 50% overlap... 0.74 20 diffes of 1k-items sets 50% overlap... 1.07 20 simdif of 1k-items sets 50% overlap... 1.72 [alex@lancelot is]$ python -O ../bench_set.py Timing Set $Revision: 1.5 $ (Python 2.3a0 (#44, Jul 18 2002, 00:03:05) [GCC 2.96 20000731 (Mandrake Linux 8.2 2.96-0.76mdk)]) Make 20 10k-items sets (no reps)... 0.21 Make 20 1k-items sets (x10 reps)... 0.36 Make 20 more 1k-items sets (no reps)... 0.38 20 unions of 1k-items sets 50% overlap... 0.42 20 inters of 1k-items sets 50% overlap... 0.75 20 diffes of 1k-items sets 50% overlap... 1.08 20 simdif of 1k-items sets 50% overlap... 1.73 [alex@lancelot is]$ but as you can see, again it's uniformly faster on both 'is' and 'has' versions of sets. The 'has' version thus seems preferable here. Alex

Oren, I like the direction this is going in, but I have some reservations about any protocol which requires users to avoid using a simple method name like next() on their own multi-pass sequence types unless they intend their sequence to be treated as single-pass. One other possibility: if x.__iter__() is x, it's a single-pass sequence. I realize this involves actually invoking the __iter__ method and conjuring up a new iterator, but that's generally a lightweight operation... -Dave From: "Oren Tirosh" <oren-py-d@hishome.net>

There is no need for a new type of iterator. It's ok that iterators are disposable. If I need multiple iterations I don't want to copy the iterator - I prefer to ask the original iterable object for a new iterator. All I need is some way to know whether the iterable object (container) can produce multiple iterators that generate the same sequence.

An object is re-iterable if its iterators do not modify its state.

The iterator of an iterator is itself. Calling the next method, by definition, modifies the internal state of an object. Therefore anything that has a next method is not re-iterable.

"hasattr(obj,'__iter__') and hasattr(obj, 'next')" is a good signature of a non re-iterable object. Unfortunately, the opposite is not true. One iterable object in Python produces iterators that modify its state when their .next() method is called - the file object.

I have just submitted a patch that makes a file into an iterator (i.e. adds a .next method to files). With this change all Python objects that have an __iter__ method and no next method produce iterators that do not modify the container. Another possibility would be to make file iterators that use seek or re-open the file to avoid modifying the file position of the parent file object. I don't think that would be a good idea because files can be devices, pipes or sockets which are not seekable.

I think it may be a good idea to add a note to the documentation pages about the iterator protocol that the iterators of a container should not modify the state of the container. If you think they must it's probably a good sign that your 'container' is not really a container and maybe it should be an iterator rather than produce iterators of itself.

Oren

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev

On Fri, Jul 12, 2002 at 06:22:05AM -0400, David Abrahams wrote:

...

Oren,

I like the direction this is going in, but I have some reservations about any protocol which requires users to avoid using a simple method name

From: "Oren Tirosh" <oren-py-d@hishome.net> like

...

next() on their own multi-pass sequence types unless they intend their sequence to be treated as single-pass.

I'm not too thrilled about it, either, but I don't think it's too bad. If you implement an object with an __iter__ method you must be aware of the iteration protocol and the next method. If you put a next method on an iterable you are most probably confusing iterators and iterables and not just using the name 'next' for some other innocent purpose.

People may have already written that innocent code, but I'm not sure the consequences of misinterpreting such sequences as single-pass are so terrible. Still, I would prefer if we were looking for "__next__" instead of next().

...

One other possibility: if x.__iter__() is x, it's a single-pass sequence. I realize this involves actually invoking the __iter__ method and conjuring up a new iterator, but that's generally a lightweight operation...

I think it is critical that all protocols should be defined by something passive like presence of attributes and attributes of attributes and not by active probing.

Isn't that passive/active distinction illusory though? What about __getattr__ methods?

I don't see how a future typing system could be retrofitted to Python otherwise (pssst, don't tell anyone, but I'm working on such a system...)

Nifty! I'd love to get a preview, if possible. Types come into play at the Python/C++ boundary, and I'm interested in how our systems will interact (c.f. http://aspn.activestate.com/ASPN/Mail/Message/types-sig/1222793) -Dave

If you put a next method on an iterable you are most probably confusing iterators and iterables and not just using the name 'next' for some other innocent purpose.

Quite to the contrary. You might have a multi-iterable class that was defined before the iterator protocol existed, and had a "built-in" iterator that keeps the iteration state in the object itself (a common design, e.g. BSD db files have this). This is OK for simple uses. But with iterators available the class might grow a proper iterator class that keeps state external from the object. But for backward compatibility reasons you cannot remove the next() method on the class itself. QED: you have a multi-iterable object that has both an __iter__ method and a next method. --Guido van Rossum (home page: http://www.python.org/~guido/)

On Fri, Jul 12, 2002 at 08:27:56AM -0400, Andrew Koenig wrote:

Oren> There is no need for a new type of iterator. It's ok that Oren> iterators are disposable. If I need multiple iterations I don't Oren> want to copy the iterator - I prefer to ask the original Oren> iterable object for a new iterator. All I need is some way to Oren> know whether the iterable object (container) can produce Oren> multiple iterators that generate the same sequence.

You are assuming that you still have access to the original iterable object. But what if all you have is an iterator? Then you need to be able to ask the iterator for a new iterator.

Here are two cases I can think of where I don't have access to the iterable object: 1. There is no iterable object. An iterator object was created directly. For example, the result of a generator function is an iterator which isn't the result of some container's __iter__ method. 2. The iterator was received as an argument and the caller sent iter(x) instead of x. In that case I guess it means that the caller doesn't *want* to give me access to x.

Oren> An object is re-iterable if its iterators do not modify its state.

Oren> The iterator of an iterator is itself. Calling the next method, Oren> by definition, modifies the internal state of an Oren> object. Therefore anything that has a next method is not Oren> re-iterable.

That's not the only possible definition of an iterator.

It isn't a definition of an iterator. It isn't even a definition of a re-iterable object, it's a sufficient (but not required) condition for objects to be re-iterable.

I'm thinking, in part, about how one might translate some of the C++ standard-library algorithms into Python.

Why not translate *what* they do instead of *how* they do it? I'm pretty sure the Python way would be shorter and simpler anyway. Oren

Oh yes -- one more use for being able to copy an iterator: If you can't copy an iterator, how can you determine the value to which the iterator refers without losing access to that value? Oren> Why not translate *what* they do instead of *how* they do it? Oren> I'm pretty sure the Python way would be shorter and simpler Oren> anyway. Maybe yes, maybe no. It would certainly be different, because the C++ algorithms generally assume that iterators support comparison operations. That assumption makes possible algorithms in C++ that are difficult to express at all using Python iterators as they stand. On the other hand, the availability of garbage collection in Python, combined with the dynamic nature of its type system, makes it possible to express algorithms in Python that cannot be expressed easily in C++ using C++ iterators as they now stand. Details about language design can have a profound effect on usage, which in turn has a profound effect on future design. -- Andrew Koenig, ark@research.att.com, http://www.research.att.com/info/ark

...

...

...

...

"AK" == Andrew Koenig <ark@research.att.com> writes:

AK> You are assuming that you still have access to the original AK> iterable object. But what if all you have is an iterator? AK> Then you need to be able to ask the iterator for a new AK> iterator. Would it be useful to add to the interator "interface" a method which would retrieve the original iterable object? I've no idea what that method should be called, but it seems like it would be trivial to add since most (all?) iterators have a pointer to their underlying object anyway, don't they? -Barry

...

...

...

...

"GvR" == Guido van Rossum <guido@python.org> writes:

>> Would it be useful to add to the interator "interface" a method >> which would retrieve the original iterable object? I've no >> idea what that method should be called, but it seems like it >> would be trivial to add since most (all?) iterators have a >> pointer to their underlying object anyway, don't they? GvR> No. The (important!) class of generator-iterators does not GvR> have an underlying container object. Yup, but in that case I think it would be fine if it.gimme_the_underlying_iteratable_object() returned None. It still may be useless. ;) -Barry

...

...

...

...

"AK" == Andrew Koenig <ark@research.att.com> writes:

Barry> Would it be useful to add to the interator "interface" a Barry> method which would retrieve the original iterable object? AK> What if there isn't one? The method would return None. -Barry

...

...

...

...

"AK" == Andrew Koenig <ark@research.att.com> writes:

Barry> Would it be useful to add to the interator "interface" a Barry> method which would retrieve the original iterable object? AK> What if there isn't one? Barry> The method would return None. AK> But then you can't rely on it. That is, if you want to write AK> code that depends on the ability to retrieve the original AK> iterable, you have to give up the ability for that code to AK> work on generators, for example. AK> I'm not saying it's not a useful thing to have; I'm just AK> saying it might not be as useful as it appears at first. I'm not sure it's even useful at all, e.g. I've never had a use for it. But if you have code that depends on the ability to retrieve the original iterable, and you have iterators for which there /is no/ original iterable, it doesn't matter how you spell it, you're going to have to special case around that fact. -Barry

On Fri, Jul 12, 2002 at 08:46:26AM -0400, Guido van Rossum wrote:

...

I have just submitted a patch that makes a file into an iterator (i.e. adds a .next method to files). With this change all Python objects that have an __iter__ method and no next method produce iterators that do not modify the container. Another possibility would be to make file iterators that use seek or re-open the file to avoid modifying the file position of the parent file object. I don't think that would be a good idea because files can be devices, pipes or sockets which are not seekable.

Cute trick, but I think it's too fragile. You don't know about 3rd party iterables that have the same problem as file.

I don't understand what you mean by fragile. I'm not suggesting anything that actually depends on this behavior so I don't see what could break. I think it's semantically cleaner for iterable objects to produce iterators that do not modify the state of the original iterable object. There's no way to force extension writers to adhere to this but Python should at least set a good example. Python file objects are not a good example. The xrange object that was its own iterator was not a good example. Oren

On Fri, Jul 12, 2002 at 09:36:22AM -0400, Guido van Rossum wrote:

...

I don't understand what you mean by fragile. I'm not suggesting anything that actually depends on this behavior so I don't see what could break.

If nothing depends on it, what's the point?

To satisfy my perverted obsession for semantic hygiene, of course!

That version of the xrange object was broken.

That's exactly my point. There will be more broken code like this as long as people keep confusing iterators and iterables. Making the language semantically cleaner should help prevent things like this in the long run. I remember it was pretty hard to actually convince anyone that xrange was broken. When I pointed out that the xrange 'iterator' modified the state of the xrange 'container' people responded that it's ok because this happens with file objects, too...

I don't see what's wrong with the file object. Iterating over a file changes the file's state, that's just a fact of life.

A file object is an iterator pretending to be a container. For historical reasons it uses 'readline' instead of 'next' and an empty string instead of StopIteration but it basically does the same job. A file object is not really a container that can produce iterators of itself. Oren

Oren Tirosh <oren-py-d@hishome.net>:

A file object is an iterator pretending to be a container. For historical reasons it uses 'readline' instead of 'next'

I think it's more complicated than that. If the file object were to become an object obeying the iterator protocol, its next() method should really return the next *byte* of the file. Then you'd still want methods like read(), readline() etc. for reading in larger chunks. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

Guido:

Me:

...

If the file object were to become an object obeying the iterator protocol, its next() method should really return the next *byte* of the file.

I don't think so. We should pick the most convenient chunking for the default iterator

But we're talking here about making the file object *be* an iterator itself, not just have a "default iterator". If that's to happen, all the other ways of iterating over a file ought to be implemented on top of the basic iteration facility provided by the file object -- lest we get unfortunate interactions between the different iteration methods a la xreadlines(). To me, this implies that the file object must iterate by bytes. I'm not necessarily advocating this, just following the idea to its logical conclusion. If the conclusion is distasteful, maybe that's a sign that the idea (i.e. making file objects into iterators) isn't so good in the first place. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

...

I don't think so. We should pick the most convenient chunking for the default iterator

But we're talking here about making the file object *be* an iterator itself, not just have a "default iterator". If that's to happen, all the other ways of iterating over a file ought to be implemented on top of the basic iteration facility provided by the file object -- lest we get unfortunate interactions between the different iteration methods a la xreadlines(). To me, this implies that the file object must iterate by bytes.

Why should all iteration over a file be defined in terms of its basic iteration? I don't see that as dogma. --Guido van Rossum (home page: http://www.python.org/~guido/)

On Fri, 12 Jul 2002, Guido van Rossum wrote:

...

I don't see what's wrong with the file object. Iterating over a file changes the file's state, that's just a fact of life.

[Ping]

That's exactly the point. Iterators and containers are different. Walking over a container shouldn't mutate it, whereas an iterator has mutable state independent of the container.

The key problem is that the file's __iter__ method returns something whose state depends on the file, thus breaking this expectation. Either __iter__ should be implemented to fulfill its commitment, or there shouldn't be an __iter__ method on files at all.

What commitment? Iterators don't have to have an undelying container! (E.g. generators.)

I'm not suggesting that the semantics of files themselves are "broken" or have a "wart" that needs to be fixed -- merely that we should decide on a place for files to live in our world of containers and iterators, so we can set and maintain consistent expectations.

What are your expectations? I think that both file.__iter__() returning file (as it does with Oren's patch) or file.__iter__() returning an xreadlines object (as it still does in CVS) are fine as far as reasonable expectations for iterators go. --Guido van Rossum (home page: http://www.python.org/~guido/)