Callable generators (PEP 288: Generator Attributes, again)

Bengt Richter bokr at
Wed Nov 19 23:28:53 CET 2003

On 19 Nov 2003 10:28:27 -0800, michele.simionato at (Michele Simionato) wrote:

>"Francis Avila" <francisgavila at> wrote in message news:<vrmpthqfk8bc8b at>...
>> I'm suggesting the PEP's functionality, not its syntax and semantics.  My
>> contention is that the PEP regards generators as too class-like, when they
>> are more naturally considered as function-like.
>> For example, your iterator class/instance would look like this:
>> def iterator(x=1)(x):
>>     yield x
>>     yield x
>> print # -> 1
>> print iterator(5)  # -> 5
>> The local name "x" is updated (according to the second parameter list in the
>> function definition) right after the yield of the previous call when
>> iterator is called, behaving like a state-persistent callable function.  If
>> it's just "nexted", it behaves like a plain old iterator.
>I see what you mean, now. Still, the notation is confusing, since I
>do think an iterator as something which it obtained by "instantiating"
>a generator. On the other hand, your iterator(5) would not be a newly
>"instantiated" iterator, but the same iterator with a different
>parameter x. So, you would need a different syntax, as for instance
>iterator[5]. Still, I do think this would be confusing. The class
>solution would be more verbose but clearer.

I really like generators, but I am not really happy with the magic transmogrification
of an ordinary function into a generator-instance factory as a magic side effect
of defining a state machine with yields in the function body. Sorry to complain
about a past decision, but one effect was also to exclude yield-ing in a subroutine
of the generator (which would in general require suspending a stack of frames, but would
open a lot of possibilities, ISTM).

But, trying to live with, and extend, current realities, we can't call the generator
factory function name again to pass values into the generator, since that merely creates
another generator. But if we have

    gen = factory(some_arg)

where factory could be e.g., (I'm just using 'factory' instead of 'iterator' for semantics)

    def factory(x=1): yield x; yield x

we could potentially call something other than, i.e., if gen had a gen.__call__ defined,


could be a way to pass data into the generator. But what would it mean w.r.t. backwards compatibility
and what would be different? Obviously the calls to generated by

   for x in factory(123): print x

would have to work as now. But consider, e.g.,

   for x in itertools.imap(factory(), 'abc'): print x

presumably it would be equivalent to

   gen = factory()
   for x in itertools.imap(gen, 'abc'): print x

and what we'd like is to have some local variables (just x in this case) set to whatever
is passed through gen.__call__(...) arg list, in this case successively 'a', 'b', and 'c'.

Since gen is the product of a factory (or implicit metaclass operating during instantiation
of the function??) it should be possible to create a particular call signature for gen.__call__.

One syntactic possibility would be as Francis suggested, i.e.,

    def iterator(x=1)(x): 
        yield x
        yield x

but, BTW, I guess Francis didn't mean

    print # -> 1
    print iterator(5)  # -> 5

since it is not iterator that has the .next method, but rather the returned
generator. So unless I misunderstand it would have to be

    gen = iterator()
    print # -> 1
    print gen(5)  # -> 5

IIRC, this is what I was getting at originally (among other things;-)

If we live with the current parameter list definition mechanism, we are forced
to pass default or dummy values in the initial factory call, even if we intend
to override them with the very first call to gen('something'). We have the option
of using them through next(), but we don't have to. I.e.,

    gen = iterator()
    print gen('hoo') # -> hoo (default 1 overridden)
    print gen('haw') # -> haw

also, incidentally,

    gen = iterator()
    print gen('hoo') # -> hoo (default 1 overridden)
    print # -> hoo (x binding undisturbed)

That doesn't seem so hard to understand, if you ask me. Of course,
getting gen.__call__ to rebind the local values requires real changes,
and there are some other things that might make usage better or worse
depending -- e.g., how to handle default parameters. On the call to the
factory, defaults would certainly act like now, but on subsequent gen() calls,
it might make sense to rebind only the explicitly passed parameters, so the default
would effectively be the existing state, and you could make short-arg-list calls. Hmmm...

This might be a good combination use of defaults in the factory call to set initial
defaults that then are optionally overridden in the gen() calls. I kind of like that ;-)

>> Here's an efficient reversable generator:
>> def count (n=0)(step=1):
>>     while True:
>>         yield n
>>         n += step

My way would be

   def count (step=1, count=0):
       while True:
           yield n
           n += step

   ctr = count()
   ctr()   # -> 0
   ctr()   # -> 1
   ctr()   # -> 2
   ctr(-1) # -> 1
   ctr()   # -> 0
   ctr()   # -> -1

>Yeah, it is lightweight, but a class derived from "Iterator" 
>would be only one line longer, so I am not convinced. Also,
>people would start thinking that you can define regular
>functions with two sets of arguments, and this would generate
>a mess ...

Well, what do you think of the above?
BTW, thanks Francis for the citation ;-)

PS, I wonder if a co_yield would be interesting/useful? (Think PDP-11 jsr pc,@(sp)+ ;-)

Bengt Richter

More information about the Python-list mailing list