[Python-Dev] PEP 340 -- loose ends

[Guido van Rossum]

[Delaney, Timothy]
> What I meant is that there are no examples of how to
> actually implement the correct semantics for a normal iterator. Doing it
> right is non-trivial, especially with the __next__ and __exit__
> interaction (see below).

Depends on what you mean by right. Ignoring the argument to __next__()
and not implementing __exit__() seems totally "right" to me.

[...]
> What I meant is how the iterator is meant to handle the parameters
> passed to each method. PEP 340 deals with this by stating that
> exceptions will be raised at the next yield-statement or -expression. I
> think we need an example though of how this would translate to a
> "normal" iterator. Something along the lines of::
> 
>     class iterator (object):
> 
>         def next (self):
>             return self.__next__()
> 
>         def __next__(self, arg=None):
>             value = None
> 
>             if isinstance(arg, ContinueIteration):

Oops. Read the most recent version of the PEP again. __next__()
doesn't take an exception argument, it only takes a value. Maybe this
removes your concern?

>                 value = arg.value
>             elif arg is not None:
>                 raise arg
> 
>             if value is None:
>                 raise StopIteration
> 
>             return value

That's a very strange iterator; it immediately terminates unless you
call __next__() with a non-None argument, then it returns the argument
value. I'm having a hard time understanding what you meant to say.
Also note that the very *first* call to __next__() is not supposed to
have an argument. The argument (normally) only comes from "continue
EXPR" and that can only be reached after the first call to __next__().
This is exactly right for generators -- the first __next__() call
there "starts" the generator at the top of its function body,
executing until the first yield is reached.

>         def __exit__(self, type=None, value=None, traceback=None):
>             if (type is None) and (value is None) and (traceback is None):
>                 type, value, traceback = sys.exc_info()

You shouldn't need to check for traceback is None.

Also, even though the PEP suggests that you can do this, I don't see a
use case for it -- the translation of a block-statement never calls
__exit__() without an exception.

>             if type is not None:
>                 try:
>                     raise type, value, traceback
>                 except type, exc:
>                     return self.__next__(exc)
> 
>            return self.__next__()

Ah, here we see the other misconception (caused by not reading the
most recent version of the PEP). __exit__() shouldn't call __next__()
-- it should just raise the exception passed in unless it has
something special to do.

Let me clarify all this with an example showing how you could write
"synchronized()" as an iterator instead of a generator.

class synchronized:
    def __init__(self, lock):
        self.lock = lock
        self.state = 0
    def __next__(self, arg=None):
        # ignores arg
        if self.state:
            assert self.state == 1
            self.lock.release()
            self.state += 1
            raise StopIteration
        else:
            self.lock.acquire()
            self.state += 1
            return None
    def __exit__(self, type, value=None, traceback=None):
        assert self.state in (0, 1, 2)
        if self.state == 1:
            self.lock.release()
        raise type, value, traceback

> >> As another option, it might be worthwhile creating a base iterator type
> >> with "correct" semantics.
> 
> > Well, what would the "correct" semantics be? What would passing a
> > parameter to a list iterator's __next__() method mean?
> 
> Sorry - I meant for user-defined iterators. And the correct semantics
> would be something like the example above I think. Except that I think
> most of it would need to be in a separate method (e.g. _next) for base
> classes to call - then things would change to be something like::
> 
>     class iterator (object):
>         ...
> 
>         def _next (self, arg):
>             if isinstance(arg, ContinueIteration):
>                 return arg.value
>             elif arg is not None:
>                 raise arg
> 
>         def __next__(self, arg=None):
>             value = self._next(arg)
> 
>             if value is None:
>                 raise StopIteration
> 
>             return value
> 
>         ...

I think this is all based on a misunderstanding of the PEP.

Also, you really don't need to implement __exit__() unless you have
some cleanup to do -- the default behavior of the block translation
only calls it if defined, and otherwise simply raises the exception.

> Finally, I think there is another loose end that hasn't been addressed::
> 
>     When __next__() is called with an argument that is not None, the
>     yield-expression that it resumes will return the value attribute
>     of the argument.  If it resumes a yield-statement, the value is
>     ignored (or should this be considered an error?).  When the
>     *initial* call to __next__() receives an argument that is not
>     None, the generator's execution is started normally; the
>     argument's value attribute is ignored (or should this be
>     considered an error?).  When __next__() is called without an
>     argument or with None as argument, and a yield-expression is
>     resumed, the yield-expression returns None.

Good catch.

> My opinion is that each of these should be an error.

Personally, I think not using the value passed into __next__() should
not be an error; that's about the same as not using the value returned
by a function you call. There are all sorts of reasons for doing that.
In a very early version of Python, the result of an expression that
wasn't used would be printed unless it was None (it still does this at
the interactive prompt); this was universally hated.

I agree that calling the initial __next__() of a generator with a
non-None argument should be considered an error; this is likely caused
by some kind of logic error; it can never happen when the generator is
called by a block statement.

I'll update the PEP to reflect this.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)