[Python-Dev] Re: Reiterability

Guido van Rossum guido at python.org
Sat Oct 18 15:55:48 EDT 2003

> >OK, I think I understand what you're after.  The code for an iterator
> >expression has to create a generator function behind the scenes, and
> >call it.  For example:
> >
> >   A = (f(x) for x in S)
> >
> >could be translated into:
> >
> >   def gen(seq):
> >       for x in seq:
> >           yield f(x)
> >   A = gen(S)
> >
> >(Note that S could be an arbitrary expression and should be evaluated
> >only once.  This translation does that correctly.)
> Interesting.  That wasn't the semantics I envisioned.  I was thinking 
> (implicitly, anyway) that an iterator comprehension was a closure.  That 
> is, that S would be evaluated each time.

We must be miscommunicating.  In

  A = [f(x) for x in S]

I certainly don't expect S to be evaluated more than once!

Did you mean "each time through the loop" or "each time we reach this
statement" or "each time someone loops over A" ???

Also note that I was giving the NON-reiterable semantics.  I don't
think there's any other way to do it (of course 'gen' should be an
anonymous function).

> However, if S is a sequence, you 
> don't need to reevaluate it, and if S is another iterator expression that 
> preserves reiterability, you still don't need to.  So, in that sense 
> there's never a need to
> >This allows one to iterate once over A (a generator function doesn't
> >allow reiteration).  What you are asking looks like it could be done
> >like this (never mind the local names):
> Yes, that's actually what I said, but I guess I was once again unclear.
> >   def gen(seq):
> >       for x in seq:
> >           yield f(x)
> >   class Helper:
> >       def __init__(seq):
> >           self.seq = seq
> >       def __iter__(self):
> >           return gen(self.seq)
> >   A = Helper(S)
> >
> >Then every time you use iter(A) gen() will be called with the saved
> >value of S as argument.
> Yes, except of course Helper would be a builtin type.

Sure, and its constructor would take 'gen' as an argument:

  class Helper:
      def __iter__(self, seq, gen):
          self.seq = seq
          self.gen = gen
      def __iter__(self):
          return self.gen(self.seq)

  def gen(seq):
      for x in seq:
          yield f(x)
  A = Helper(S, gen)

> >I don't mind that so much, but I don't think all the extra machinery
> >is worth it; the compiler generally can't tell if it is needed so it
> >has to produce the reiterable code every time.
> It has to produce the generator every time, anyway, presumably as a
> nested function with access to the current locals.  The only
> question is whether it can be invoked more than once, and whether
> you create the helper object.  But maybe that's what you mean, and
> now you're being unclear instead of me.  ;)

I meant creation of the Helper instance.  Given that in most practical
situations if you *need* reiterability you can provide it using
something much simpler, I don't like using a Helper instance.

But in fact I don't even like having the implicit generator function.
I guess that's one reason I'm falling down on the -0 side of this

> >   If you *want* to
> >have an iterable instead of an iterator, it's usually easy enough do
> >(especially given knowledge about the type of S).
> I just tend to wish that I didn't have to think about whether
> iterators are reiterable or not, as it forces me to expose to
> callers of a function whether the value they pass must be an
> iterator or an iterable.

To me that's a perfectly reasonable requirement, as long as functions
taking an iterator also take an iterable (i.e. they call iter() on
their argument), so a caller who has only iterables doesn't have to
care about the difference.

--Guido van Rossum (home page: http://www.python.org/~guido/)

More information about the Python-Dev mailing list