[Python-3000] Builtin iterator type

[George Sakkis]

On 11/15/06, Iain Lowe <me at ilowe.net> wrote:

> Hi George,
>
> I've been reading this thread with some interest since you seem to be
> really adamant about the "features" you are requesting. I went back into
> the c.l.p thread and did a little more searching before finding this
> thread: http://shrunklink.com/?nnq

Glad you ran a full investigation on me before replying, that's reassuring :-)

> I don't really understand how you can be advocating the opposite of what
> you explain rather clearly in the referenced thread. Specifically you
> say "there's no particular reason this should be a method of class A
> since it can be used for any arbitrary object with no extra overhead.
> Now, if you intend to use it only for instances of A and its subclasses,
> the only difference would be syntactic."
>
> Now, clearly we are not talking about "any arbitrary object" but rather
> "any arbitrary object that implements the iterable protocol" but it
> seems to me that there is little difference between what you are
> proposing here and the OOP solution the poster in the referenced thread
> was proposing.
>
> In both cases, the implementation of a series of constraints as an
> *interface* instead of as a *protocol* un-necessarily limits future
> code. It also seems clear to me that you would like to be able to
> slice/chain/etc. *any* iterable, not just instances of (subclasses of)
> `Iter`. So why advocate the narrowing of the *protocol* into an
> *interface* just to add some syntax sugar. Indeed, the two most-quoted
> examples so far (slicing and indexing) can be better written as
> "list(it)[:3]" and "list(it)[6]". The only possible caveat I can see is
> that this will not work for infinite iterables. But *even if* you did
> implement this "class-wrapper" for iterables you *still* would be
> breaking the implementation of slices (since you would not be able to do
> "Iter(infinite_it)[-5]").
>
> What am I missing here?

Several things:

1. In the thread you refer to, the whole point was *one* callable
(listMethods) that takes a single arbitrary object. I wouldn't propose
a class for holding a single method of course, but here we're talking
about a dozen or so callables that take an iterable (and optionally
other parameters).

2. If one wanted to attach listMethods to an existing type, the only
obvious choice would be the base object type. That callable isn't so
important though to pollute the base object namespace with a new
method. OTOH in this thread:
  (1) I don't propose to pollute object or any other existing type; I
propose a brand new "rich iterator" type, and
  (2) I consider several itertools functions (chain, islice, izip,
enumerate) fundamental in getting the most out of iterables.
Surprizingly, as it turns out nobody else here seems to share this
feeling. I'd say this is the only argument so far I have nothing to
respond to. If people consider itertools a small obscure module they
rarely need to import anyway, the whole proposal fails flat.

3. I don't know if you've followed the thread for the last day or so,
but I've dropped the part that suggested the proposed type as a
required base for all iterators. Instead I suggest it as a wrapper (or
"adaptor" if you prefer) to provide a rich-iterator API to the code
that needs it. The overhead of wrapping an arbitrary iterable is
trivial, both in keystrokes ("Iter(x)") and in runtime, so objections
of the kind "how should a client know whether he is passed a simple
iterator or a rich one?" are easily responded by "never assume you are
passed a rich iterator; just wrap it with Iter() yourself if you need
it".

4. As for the negative indices that don't work for iterators, let me
remind you that indexing and slicing, like pretty much everything else
in python, are an informal API. Numpy array slices also have different
semantics from list slices, and for good reason. Another hypothetical
example would be an automatically extensible sequence type defined so
that s[i] for i>=len(s) doesn't raise IndexError, but it extends the
sequence with some default value up to i and the returns s[i]. The
bottom line is, you can't assume you know what x[:5] does without
knowing x's type, the same way you don't if x.shoot() will shoot a
photo or a bullet without knowing what x is.

> P.S. I notice in a couple of places you ask what the objections are
> whereas I think that in this case it's up to the proposal's author to
> explain why the community can't live without the feature rather than for
> the community to explain why they don't need it. Clearly if they needed
> it they would have built it before (or at least you would just get a
> stream +1's)

Right, and that's the part I mistakenly took for granted, the
prevalence of itertools. My bad.

George