[Python-Dev] Return from generators in Python 3.2

[Yury Selivanov]

Hello,

I want to bring up a "forbidden" topic, however, I believe I have some strong points.

There are many ways of doing asynchronous programming in Python.  Multiprocessing, 
Threads, Greenlets, Deferred Object (Callbacks) and Coroutines.  The latter is quite 
a new approach, but it gets more and more attention.  What's really fascinating about 
coroutines, is that a code flow of a program using them reads naturally and straight.  
Callbacks break that code flow making it much harder to read and understand, threads 
don't work good in Python, and greenlets... greenlets  are too magical, and,
potentially, harmful.

So, coroutines are good, and that is proved by a pleiad of new frameworks that utilize
them: Monocle, cogen, and many others. However, coroutines in python are a bit 
incomplete.  There is no standard way of returning a result value, making coroutine
stop.

Let's take a look at the following example:

... @bus.method
... def method1():
...		# some computation
...		return result
...
... @bus.method
... def method2():
...		data = yield memcache.get(...)
...     # some computation
...
...     # and now, the most interesting point. Time to return a result. 
...		# Pick the prettiest line:
...		#      
...		# yield Return(result)
...		# return_ (result)
...		# raise StopIteration(result)

As you can see, there is no way of simple abstraction of coroutines.  How nice is
the 'yield' syntax here, that clearly marks async call, and how ugly is the return 
code.  Speaking about large amounts of a code like the above it's hard to maintain 
and refactor it.  Adding one yield statement to some generically decorated handler 
will force you to fix all returns and vice versa.  Moreover, lack of proper return 
protocol complicates the underlying code.

The very straightforward solution was proposed in PEP 380, and  here it is a good 
place to say, that PEP 380 is not all about returns.  It's all about new 'yield from' 
statement, and the new return syntax for coroutine is the very small part of it.
However, in any currently existing framework it is possible to implement 'yield from' 
statement (with smth like yield From(...)), but there's absolutely no  way to correct 
the return problem, as it raises SyntaxError which is impossible to catch.  Therefore, 
I think that we can consider the returns problem apart from PEP 380.

Proposed change uses the same type of approach as was introduced in PEP 380, but in
a slightly different way.  Instead of attaching the return value to StopIteration
exception, we can introduce another one, let's call it GeneratorReturn (derived from
BaseException).  Still easy to use it in frameworks, but make it impossible to break
things unintentionally.  For example, it will protect us from cases like the 
following:

... def test():
...		for i in range(10):
...			yield i
...		return 10

In the above, GeneratorReturn error will be propagated stopping the program execution.
Strictly speaking, the proposed change is just alters the current Python behaviour,
making the 'return value' statement raise catchable error (instead of SyntaxError.)

Speaking about PEP 3003.  I'm pretty much sure that the idea behind moratorium on
serious language changes was to give alternative python interpreters a chance to
catch up Python 3.  Well, the proposed is a very small change in CPython, just few
lines of code.  It doesn't change grammar or AST tree structure, and it is fully
backwards compatible.  I've looked at the PyPy code and found that the change is 
*very* easy to port there, and I'm certain that the situation is the same for Jython 
and IronPython.  (If this new feature would be the only problem why we don't see
Jython or PyPy supporting 3.2 version we all would be more than happy.)  Given all
that, I think PEP 3003 is inapplicable to this proposal.


Pros:
 - The change on the interpreter side is tiny (reducing the entropy in symtable.c!)
 - No affect on grammar or AST structure.
 - Easy to port to other interpreters.
 - Fully backward compatible.
 - On the very basic level it will change current behaviour from raising an 
   uncatchable error to raising a catchable one.  Nobody will be confused.
 - Another key feature of Python 3, that will probably encourage people to
   migrate.
 - Will make coroutines more attractive and stimulate the rise of new frameworks
   and development of new ones.
 - One way of doing things.  The same interface in frameworks, code in coroutines
   look almost the same as in subroutines but with yields.  Make coroutines protocol
   complete.


If we decide to postpone this feature till Python 3.3, than we'll push it all back
for *years*.  The change is tiny, but it means really a lot. Those who tried to 
work with coroutines will understand me.  Let's at least consider it.


PS I'm attaching a patch to the letter; it's far from ideal state, but contains the
GeneratorReturn exception, code to raise it and the corresponding unittests.


-
Yury

-------------- next part --------------
A non-text attachment was scrubbed...
Name: generators_return.patch
Type: application/octet-stream
Size: 7179 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100826/a11f5933/attachment.obj>