[Python-Dev] PEP 479: Change StopIteration handling inside generators

[Raymond Hettinger]

> On Nov 22, 2014, at 2:45 PM, Chris Angelico <rosuav at gmail.com> wrote:
> 
> Does your middleware_generator work with just a single element,
> yielding either one output value or none?


I apologize if I didn't make the point clearly.  The middleware example was 
just simple outline of calling next(), doing some processing, and yielding a
result while letting the StopIteration float through from the next() call.

It was meant to show in summary form a pattern for legitimate uses of next() 
inside a generator.   Some of those uses benefit from letting their StopIteration
pass through rather than being caught, returning, and reraising the StopIteration.

The worry is that your proposal intentionally breaks that code which is currently
bug free, clean, fast, stable, and relying on a part of the API that has been
guaranteed and documented from day one.

Since the middleware() example was ineffective in communicating the need,
here are some real-world examples.

Here's one from Fredrick Lundh's ElementTree code in the standard library
(there are several other examples besides this one in his code are well):

    def iterfind(elem, path, namespaces=None):
        # compile selector pattern
        cache_key = (path, None if namespaces is None
                                else tuple(sorted(namespaces.items())))
        if path[-1:] == "/":
            path = path + "*" # implicit all (FIXME: keep this?)
        try:
            selector = _cache[cache_key]
        except KeyError:
            if len(_cache) > 100:
                _cache.clear()
            if path[:1] == "/":
                raise SyntaxError("cannot use absolute path on element")
            next = iter(xpath_tokenizer(path, namespaces)).__next__
            token = next()
            selector = []
            while 1:
                try:
                    selector.append(ops[token[0]](next, token))
                except StopIteration:
                    raise SyntaxError("invalid path")
                try:
                    token = next()
                    if token[0] == "/":
                        token = next()
                except StopIteration:
                    break
            _cache[cache_key] = selector
        # execute selector pattern
        result = [elem]
        context = _SelectorContext(elem)
        for select in selector:
            result = select(context, result)
        return result

And here is an example from the pure python version of one of the itertools:

    def accumulate(iterable, func=operator.add):
        'Return running totals'
        # accumulate([1,2,3,4,5]) --> 1 3 6 10 15
        # accumulate([1,2,3,4,5], operator.mul) --> 1 2 6 24 120
        it = iter(iterable)
        total = next(it)
        yield total
        for element in it:
            total = func(total, element)
            yield total

And here is an example from Django:

    def _generator():
        it = iter(text.split(' '))
        word = next(it)
        yield word
        pos = len(word) - word.rfind('\n') - 1
        for word in it:
            if "\n" in word:
                lines = word.split('\n')
            else:
                lines = (word,)
            pos += len(lines[0]) + 1
            if pos > width:
                yield '\n'
                pos = len(lines[-1])
            else:
                yield ' '
                if len(lines) > 1:
                    pos = len(lines[-1])
            yield word
    return ''.join(_generator())

I could scan for even more examples, but I think you get the gist.
All I'm asking is that you consider that your proposal will do more
harm than good.  It doesn't add any new capability at all.
It just kills some code that currently works.


Raymond
(the author of the generator expressions pep)



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20141122/56fa9e66/attachment.html>