Enhanced Generators - the difference between sources and iterators

[Kragen Sitaker]

Oren Tirosh <oren-py-l at hishome.net> writes:
> On Fri, Feb 01, 2002 at 07:51:01PM -0500, Kragen Sitaker wrote:
> > Oren Tirosh <oren-py-l at hishome.net> writes:
> > > My implementation of xmap, xfilter and xzip returns a source, not an
> > > iterator.  It can be iterated multiple times, just like the output of map,
> > > filter or zip.
> > 
> > That won't work when passed iterators as arguments, right?
> 
> Sure, but as long as iterators are kept as invisible temporary objects

Um, sure.  iter(open("hello")) doesn't work that way already.
Iterators are not required to be restartable, and their greatest
usefulness is when they *aren't* invisible temporary objects ---
e.g. in a parsing algorithm.

In any case, it isn't clear to me that it's more useful to be able to
say:
    mymap = xmap(f, xs)
    for ii in zs:
	for jj in mymap:
	    do_something(ii, jj)
than to have to say:
    for ii in zs:
	for jj in xmap(f, xs):
	    do_something(ii, jj)

Which is what restartable xmap etc. gives you.  Right?  I think you
can generally make nonrestartable xmap() etc. restartable (assuming
their arguments are) with a lambda:
    mymap = lambda: xmap(f, xs) # assumes nested scopes
    for ii in zs:
        for jj in mymap():
            do_something(ii, jj)
So the upside of restartable xmap() etc. is that it saves you a lambda
and a lambda call when you need it; the downside is that it gives you
a function whose semantics subtly break when it's applied to
nonrestartable iterators.

> > Having "sinks" just gives you another way to do the same thing.  
> 
> You are correct in theory, but not in practice.  In the course of a 
> dataflow it can switch only once from pull to push without the use
> of threads or queues (try it!).  I like generators because they let you 
> go much further without resorting to the use of threads.  (Thanks Guido!).
> ...
> Sure, the two forms are equivalent.  But *using* them is very different.
> One of them exposes a push-mode interface and the other exposes a pull-mode
> interface.  Try to write code that wraps one of them so that it exposes 
> the other type of interface.  You'll soon see what I mean.

That's a serious objection I hadn't thought of --- adding the "accept"
feature to the language wouldn't add any expressive power, but it
would introduce gratuitous incompatibility between code written with
one mechanism and code written with the other, which would be very
hard to bridge.  For example, you could write a general 'uniq'
transformation for iterators that would work in 'pull' mode, and you'd
basically have to cut-and-paste the code and make some changes to get
it to work with a sink in 'push' mode.  That sounds like a really good
idea not to consider adding this feature to Python!

If you want to justify its addition to the language, you need to
demonstrate some case where it adds expressive power --- which means
showing some code, even toy code, that becomes simpler or easier to
understand when you add it.