[Python-Dev] Return from generators in Python 3.2

[Yury Selivanov]

On 2010-08-26, at 8:25 PM, Guido van Rossum wrote:

> On Thu, Aug 26, 2010 at 5:05 PM, Yury Selivanov <yselivanov at gmail.com> wrote:
>> On 2010-08-26, at 8:04 PM, Greg Ewing wrote:
>>> Even with your proposal, you'd still have to use a 'creepy
>>> abstraction' every time one of your coroutines calls another.
>>> That's why PEP 380 deals with 'more than just return'.
>> 
>> Nope.  In almost any coroutine framework you have a scheduler
>> or trampoline object that basically does all the work of calling,
>> passing values and propagating exceptions.  And many other things
>> that 'yield from' won't help you with (cooperation, deferring to
>> process/thread pools, pausing, etc.)  Being a developer of one
>> of such frameworks, I can tell you, that I can easily live without
>> 'yield from', but dealing with weird return syntax is a pain.
> 
> That's not my experience. I wrote a trampoline myself (not released
> yet), and found that I had to write a lot more code to deal with the
> absence of yield-from than to deal with returns. In my framework,
> users write 'raise Return(value)' where Return is a subclass of
> StopIteration. The trampoline code that must be written to deal with
> StopIteration can be extended trivially to deal with this. The only
> reason I chose to use a subclass is so that I can diagnose when the
> return value is not used, but I could have chosen to ignore this or
> just diagnose whenever the argument to StopIteration is not None.

In the framework I'm talking about (not yet released too, but I do plan
to open source it one day), everything that can be yielded is an
instance of a special class - Command.  Generators are wrapped with a 
subclass of Command - Task, socket methods return Recv & Send commands
etc.  Regular python functions, by the way, can be also wrapped in a 
Task command, - the framework is smart enough to manage them automatically.
Of course, wrapping of python functions is abstracted in decorators,
so be it a simple @coroutine, or some asynchronous @bus.method - they
all are native objects to the scheduler.  And all the work of exception
propagation, IO waiting, managing of timeouts and much more is a business
framework (without 'yield from'.)

Hence the yield statement is nothing more than a point of a code flow where 
some command is pushed to the scheduler for execution.  And can be inserted
almost everywhere.

This approach differs from the one you showed in PEP 342; it's much more
complicated, but is has its own strong advantages.  It is not new though,
for instance, almost the same idea is utilized in 'cogen' framework, and
few others (can't remember all names but I did quite a big research before
writing a single line of code.)  All those frameworks are suffer from the
inability of using native return statement in generators.

Now, imagine a big project.  I mean really big complicated system, with
tens of thousands lines of code.  Code is broken down to small methods:
some of them implement some asynchronous methods on a message bus, 
some of them are mapped to serve responses on specific URLs and so on.
In the way of writing code I'm talking about, there is no distinction
between coroutines and subroutines.  There are some methods which just
return some value; some that query a potentially blocking code with 
'yield' keyword and after that they return the result - it all doesn't
matter.  Abstraction is very good and simple, 'yield' statement just marks
suspension points, and thats all.  BUT - there is a 'return problem',
so if the code got a new yield statement - you have to go and fix all 
returns and vice versa.  It just *breaks the beauty* of the language.
I've invested tons of time into it, and suffer from the weird syntax that
differs from one line to another.

Of course I can live with that, and people that developed other frameworks
will too.  But considering that the 'return' syntax is almost approved (1);
almost one hundred percent it will be merged to 3.3 (2); the change is small
and backwards compatible (3); one-two hours of work to port to other 
interpreters - so not contradict 100% with the moratorium ideas (4)  
- I've decided to bring this topic up.

The asynchronous programming is booming now.  It gets more and more 
attention day by day.  And python has a unique combination of features
that may make it one of the leaders in the field (nodejs is amateur;
erlang is hard; java, ruby and family lacks 'yield' statement so you
have to use callbacks - and that's ugly.)  Wait for this simple
feature for several years in a world that is changing that fast?
I'm not sure.

Probably the last point - this would be one more good advantage of py3k 
for python 2.x users.

Sorry for such a long text, I just wanted to make my points clear and 
provide some examples.

>> Especially when you use decorators like @bus.method, or
>> @protocol.handler, that transparently wrap your callable be it
>> generator or regular function.  And after that you have to use
>> different return syntax for them.
> 
> Until PEP 380 is implemented, you have to use different return syntax
> in generators. You have some choices: raise StopIteration(value),
> raise SomethingElse(value), or callSomeFunction(value) -- where
> callSomeFunction raises the exception. I like the raise variants
> because they signal to tools that the flow control stops here -- e.g.
> in Emacs, python-mode.el automatically dedents after a 'raise' or
> 'return' but not after a call (of course).

I'm not asking for the whole PEP380, but for a small subset of it.  So if
it's not that much contradicts with moratorium - let's discuss the feature.
If it is - then OK, I stop spamming ;-)

-
Yury