[Python-ideas] Exceptions thrown from generators.. patch.

[Gregory P. Smith]

On Fri, Nov 18, 2011 at 10:24 PM, Ron Adam <ron3200 at gmail.com> wrote:

>
> I was able to create a patch for testing this idea.  The hard part was
> in getting to know cpython well enough to do it.  :-)
>
>
> To get it to work, I made the following change in ceval.c so that the
> main loop will accept a pointer rather than an int for the throwflag.
> That allows an opcode to set it before it yields an exception.  ie... a
> reverse throw.
>
> PyObject *
> PyEval_EvalFrameEx(PyFrameObject *f, int throwflag) {
>    return PyEval_EvalFrame_Ex(f, &throwflag);
> }
>
> PyObject *
> PyEval_EvalFrame_Ex(PyFrameObject *f, int *throwflag)
> {
>   ...
>        TARGET(YIELD_EXCEPT)
>            *throwflag = 1;
>            retval = POP();
>            f->f_stacktop = stack_pointer;
>            why = WHY_YIELD;
>            goto fast_yield;
>   ...
>
>
> The genobject gen_send_ex() function checks the throwflag value after a
> send to see if it got a thrown out exception back.  (Rather than one
> yielded out.)
>
> A new keyword 'throws' was needed to go with the YIELD_EXCEPT opcode. I
> didn't see anyway to do it with a function or method.  It should be
> possible to set the exception in the ceval loop rather than yielding it
> out, but I think that would be more complex.
>
> Because the exception isn't raised inside the generator code object, but
> is yielded out first, the generator can be continued as if it was a
> regular yielded value.  No magic required, and no fiddling with the
> exception stack was needed. It doesn't effect unexpected exceptions, or
> exceptions raised with raise.  Those will terminate a generator as
> always.
>
> Python 3.3.0a0 (qbase qtip tip yield_except:92ac2848438f+, Nov 18 2011,
> 21:59:57)
> [GCC 4.6.1] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> >>> def G():
> ...     yield 1
> ...     throws ValueError
> ...     yield 2
> ...
> >>> g = G()
> >>> next(g)
> 1
> >>> next(g)
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
> ValueError
> >>> next(g)
> 2
> >>> next(g)
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
> StopIteration
>
>
> The three main benefits of being able to do this are...
>
> * To use switch like exception structures for flow control in schedulers
> and coroutines.
>
> * For consumer type coroutines to be able reject and re-request data in
> a nice way without terminating.   ie.. the reverse of throwing in an
> exception in, in order to change what a generator does.
>
> * It creates alternative channels for data input and output by being
> able to both throw exceptions in and out of generators.  Those signals
> can carry objects in and out and not burden the fast yield data path
> with testing for special wrapper objects.
>
>
> Here's an example of it being used in a simple scheduler.
> -----------
>
> class Suspend(Exception): pass
>
> def Person(name, count, mode):
>    n = 0
>    while n < count:
>        if mode == 0:
>            # The normal data path.
>            yield name, count
>        else:
>            # Use an exception as an alternative data path.
>            throws Suspend(name, count)
>        n += 1
>    # return
>    raise StopIteration(name, n)
>
> def main(data, mode):
>    stack = [Person(*(args + (mode,))) for args in data]
>    results = []
>    while stack:
>        done = []
>        for ct in stack:
>            try:
>                print('yield', *next(ct))     # from yield
>            except Suspend as exc:
>                print('throws', *exc.args)    # from throws
>            except StopIteration as exc:
>                results.append(exc.args)
>                continue
>            done.append(ct)
>        stack = done
>    print(results)
>    return results
>
> if __name__ == "__main__":
>    data = [("John", 2), ("Micheal", 3), ("Terry", 4)]
>    results1 = main(data, 0)
>    results2 = main(data, 1)
>    assert(results1 == results2 == data)
>
> -------------
> The output looks like...
>
> yield John 2
> yield Micheal 3
> yield Terry 4
> yield John 2
> yield Micheal 3
> yield Terry 4
> yield Micheal 3
> yield Terry 4
> yield Terry 4
> [('John', 2), ('Micheal', 3), ('Terry', 4)]
> throws John 2
> throws Micheal 3
> throws Terry 4
> throws John 2
> throws Micheal 3
> throws Terry 4
> throws Micheal 3
> throws Terry 4
> throws Terry 4
> [('John', 2), ('Micheal', 3), ('Terry', 4)]
>
>
> This shows that 'throws' works a lot like 'yield'.
>
>
neat!


>
> Open issues:
>
> * A better name than 'throws' might be good.
>

I don't like adding another keyword or confusing things by adding the
"throw" verb to a language that already firmly uses the verb "raise" when
speaking of exceptions.

A double word syntax might make sense here.  It'd be good to keep 'yeild'
in it to make it clear that this is offering the exception out in a
non-terminal manner.

  yield raise MyException(x,y,z)

or if you've caught an exception from something and want to pass that on to
the caller without killing the iteration the natural base form of that
would be:

  yield raise

to unset the existing exception and yeild it out instead, mirroring what a
bare raise within an except: clause does.


> * Should it get the object sent in.
>
>   <object> = throws <exception>
>
> Or should it be ...
>
>   throws <exception>
>
> * What would be the best argument form.. Should it take the same
> arguments as raise or just a single expression.
>
>
> Python's test suite passes as this doesn't change anything that already
> works.
>
> I haven't tested it with the yield-from patch yet, but I think if it can
> throw out exceptions in the same way yield-from yields out, that it will
> make some things easier and nicer to do.
>
> If anyone is interested, I can create a tracker item and put the patch
> there where it can be improved further.
>
> Cheers,
>   Ron
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111118/0c975a1d/attachment.html>