[Python-Dev] Yield-From Implementation Updated for Python 3

[P.J. Eby]

At 09:24 PM 8/1/2010 -0700, Guido van Rossum wrote:
>I don't understand all the details and corner
>cases (e.g. the concatenation of stacks

It's just to ensure that you never have From's iterating over other 
From's, vs. just iterating whatever's at the top of the stack.


>which seems to have to do with the special-casing of From objects in __new__)

It isn't connected, actually except that it's another place where I'm 
keeping From's flat, instead of nested.  (I hear that flat is better.  ;-) )


>I am curious whether, if you need a trampoline for async I/O anyway,
>there would be a swaying argument for integrating this functionality
>into the general trampoline (as in the PEP 342 example),

Originally, that was why I wasn't very enthusiastic about PEP 380; it 
didn't seem to me to be adding any new value over what you could do 
with existing, widely-used libraries.  (Twisted's had its own *and* 
multiple third-party From-ish libraries supporting it for many years now.)

After I wrote From(), however (which was originally intended to show 
why I thought 380 was unnecessary), I realized that having One 
Obvious Way to implement generator-based pseudothreads independent of 
an event loop, is actually useful precisely *because* it separates 
the pseudothreadedness from what you're using the pseudothreadedness for.

Essentially, the PEP 380-ish bit is the hardest part of writing an 
actual pseudothread implementation; connecting that implementation to 
an I/O framework is actually the relatively simple part.  You just 
write code that steps into the generator, and uses the yielded object 
to initiate an I/O operation and register a callback.  (If you're 
using Twisted or something else that has "promise"-like deferred 
results, it's *really* easy, because you only have a couple of types 
of yielded objects to deal with, and a uniform callback signature.)

Indeed, if you're using an existing async I/O framework, you don't 
even really *have* a "trampoline" as such -- you just have a bit of 
code that registers callbacks to itself, and the app's main event 
loop just calls back to that wrapper when the I/O is done.

In effect, an I/O framework integration would just give you a single 
API like, "run(From(geniter))", which then performs one iteration, 
and then registers whatever callback it's told to by the yield, and 
the callback it registers would actually be a reinvocation of run() 
on the same From instance when the I/O is ready, but with a value to 
pass back into the send(), or an error to throw().  So, the I/O 
framework's event loop is half of the "trampoline", and the wrapper 
that sends or throws, then registers an I/O callback, is the other half.

Something like:

     def run(coroutine, value=None, exc_info=()):
         if exc_info:
             action = coroutine.throw(*exc_info)
         else:
             action = coroutine.send(value)
         action.registerCallback(partial(run, coroutine))

Where 'action' is some I/O command object, and registerCallback() 
will call its argument back with a value or exc_info, after the I/O is done.

Of course, a real framework integration might actually dispatch on 
type here rather than using special command objects like this, and 
there might be more glue code to deal with exceptions, but really, 
the heart of the thing is just going to look like that.  (I just 
wrote it that way to show the basic structure.)

Really, it's just a few functions, maybe a utility routine or two, 
and maybe a big if-then or dictionary dispatch on types if you just 
want to be able to 'yield' existing I/O objects provided by the frameworks.

IOW, it's a *lot* simpler than actually rolling your own I/O or GUI 
framework like Twisted or Eventlet or wxPython or tk or some other such thing.


>But it seems a bit of a waste to have two different trampolines,
>especially since the trampoline itself is so hard to understand
>(speaking for myself here :-). ISTM that the single combined
>trampoline is easier to understand than the From class.

Well, the PEP 342 example was made to look simple, because it doesn't 
have to actually DO anything (like I/O!)  To work for real, it'd need 
some pluggability, and some things to help it interoperate with 
different GUI and I/O frameworks and event loops.  (Using your own 
event loop "for real" isn't very useful in a lot of non-trivial applications.)

Heck, after writing From(), it gave me an idea that I could just 
write a trampoline that *could* integrate with other event loops, 
with an idea to have it be a general-purpose companion to From.

But, after several wasted hours, I realized that yes, it *could* be 
written (I still have the draft), but it was mostly just something 
that would save a little boilerplate in bolting From()'s onto an 
existing async I/O framework, and not really anything to write home about.

So, I guess what I'm saying is, the benefit of separating the 
trampoline from control flow, is that people can then use them with 
their favorite event loop or framework, instead of the stdlib trying 
to compete with the experts on a slower release cycle.

There's additional benefit here in that 1) getting pseudothread 
implementations correct can be difficult, but once done, they're 
extremely stable, and 2) having a blessed syntax for identifying 
pseudothreaded calls and returns is a boon to competition in I/O frameworks.

So, instead of everybody having their own versions of From (and I've 
written more than a couple in my time), there's One Obvious Way To Do 
It.  All that will differ between I/O libraries are the actual API 
calls for I/O and scheduling and starting up pseudothreads, so there 
won't be as big of a switching barrier between frameworks.