[Python-ideas] Exceptions thrown from generators.. patch.

Ron Adam ron3200 at gmail.com
Sat Nov 19 07:24:39 CET 2011


I was able to create a patch for testing this idea.  The hard part was
in getting to know cpython well enough to do it.  :-)


To get it to work, I made the following change in ceval.c so that the
main loop will accept a pointer rather than an int for the throwflag.
That allows an opcode to set it before it yields an exception.  ie... a
reverse throw.

PyObject *
PyEval_EvalFrameEx(PyFrameObject *f, int throwflag) {
    return PyEval_EvalFrame_Ex(f, &throwflag);
}

PyObject *
PyEval_EvalFrame_Ex(PyFrameObject *f, int *throwflag)
{
   ...
        TARGET(YIELD_EXCEPT)        
            *throwflag = 1;
            retval = POP();
            f->f_stacktop = stack_pointer;
            why = WHY_YIELD;
            goto fast_yield;
   ...


The genobject gen_send_ex() function checks the throwflag value after a
send to see if it got a thrown out exception back.  (Rather than one
yielded out.)

A new keyword 'throws' was needed to go with the YIELD_EXCEPT opcode. I
didn't see anyway to do it with a function or method.  It should be
possible to set the exception in the ceval loop rather than yielding it
out, but I think that would be more complex. 

Because the exception isn't raised inside the generator code object, but
is yielded out first, the generator can be continued as if it was a
regular yielded value.  No magic required, and no fiddling with the
exception stack was needed. It doesn't effect unexpected exceptions, or
exceptions raised with raise.  Those will terminate a generator as
always.

Python 3.3.0a0 (qbase qtip tip yield_except:92ac2848438f+, Nov 18 2011,
21:59:57) 
[GCC 4.6.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> def G():
...     yield 1
...     throws ValueError
...     yield 2
... 
>>> g = G()
>>> next(g)
1
>>> next(g)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError
>>> next(g)
2
>>> next(g)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration


The three main benefits of being able to do this are...

* To use switch like exception structures for flow control in schedulers
and coroutines.

* For consumer type coroutines to be able reject and re-request data in
a nice way without terminating.   ie.. the reverse of throwing in an
exception in, in order to change what a generator does.

* It creates alternative channels for data input and output by being
able to both throw exceptions in and out of generators.  Those signals
can carry objects in and out and not burden the fast yield data path
with testing for special wrapper objects.


Here's an example of it being used in a simple scheduler.
-----------

class Suspend(Exception): pass

def Person(name, count, mode):
    n = 0
    while n < count:
        if mode == 0:
            # The normal data path.
            yield name, count
        else:
            # Use an exception as an alternative data path.
            throws Suspend(name, count)
        n += 1
    # return
    raise StopIteration(name, n)

def main(data, mode):
    stack = [Person(*(args + (mode,))) for args in data]
    results = []
    while stack:
        done = []
        for ct in stack:
            try:
                print('yield', *next(ct))     # from yield
            except Suspend as exc:
                print('throws', *exc.args)    # from throws
            except StopIteration as exc:
                results.append(exc.args)
                continue
            done.append(ct)
        stack = done
    print(results)
    return results

if __name__ == "__main__":
    data = [("John", 2), ("Micheal", 3), ("Terry", 4)]
    results1 = main(data, 0)
    results2 = main(data, 1)
    assert(results1 == results2 == data)

-------------
The output looks like...

yield John 2
yield Micheal 3
yield Terry 4
yield John 2
yield Micheal 3
yield Terry 4
yield Micheal 3
yield Terry 4
yield Terry 4
[('John', 2), ('Micheal', 3), ('Terry', 4)]
throws John 2
throws Micheal 3
throws Terry 4
throws John 2
throws Micheal 3
throws Terry 4
throws Micheal 3
throws Terry 4
throws Terry 4
[('John', 2), ('Micheal', 3), ('Terry', 4)]


This shows that 'throws' works a lot like 'yield'.


Open issues:

* A better name than 'throws' might be good.

* Should it get the object sent in.

   <object> = throws <exception>

Or should it be ...

   throws <exception>

* What would be the best argument form.. Should it take the same
arguments as raise or just a single expression.


Python's test suite passes as this doesn't change anything that already
works.

I haven't tested it with the yield-from patch yet, but I think if it can
throw out exceptions in the same way yield-from yields out, that it will
make some things easier and nicer to do.

If anyone is interested, I can create a tracker item and put the patch
there where it can be improved further.

Cheers,
   Ron





More information about the Python-ideas mailing list