[Python-ideas] Cofunctions PEP - Revision 4

M.-A. Lemburg mal at egenix.com
Mon Aug 16 17:20:28 CEST 2010

Greg Ewing wrote:
> M.-A. Lemburg wrote:
>> Greg Ewing wrote:
>>> In an application that requires thousands of small, cooperating
>>> processes,
>> Sure, and those use Stackless to solve the problem, which IMHO
>> provides a much more Pythonic approach to these things.
> At the expense of using a non-standard Python installation,
> though. I'm trying to design something that can be incorporated
> into standard Python and work without requiring any deep
> black magic. Guido has so far rejected any idea of merging
> Stackless into CPython.

The problem with doing so is twofold:

 1. The use case Stackless addresses is not something an everyday
    programmer will need, so making CPython more complicated
    just to add this one extra feature, doesn't appear worth
    the trouble.

 2. The Stackless implementation is not very portable, so
    the feature would only be available on a limited number of

Apart from that, every new feature will raise the bar for learning

If you could turn your proposal into something more like
the Stackless tasklets and move the implementation to an
extension module (perhaps with some extra help from new CPython
APIs), then I'm sure the proposal would get more followers.

> Also I gather that Stackless works by copying pieces of
> C stack around, which is probably more lightweight than using
> an OS thread, but not as light as it could be.

Well, it works great in practice and is a proven approach.
Copying in C is certainly fast enough for most needs and
the black magic is well hidden in Stackless.

> And I'm not sure what criteria to judge pythonicity by in
> all this. 

"explicit is better than implicit".

Tasklets are normal Python objects wrapping functions.
The create of those tasklets is explicit, not implicit
via some (special) yield burried deep in the code.

> Stackless tasklets work without requiring any kind
> of function or call markers -- everything looks exactly
> like normal Python code.

Right, because everything *is* normal Python code. Tasklets
are much more like threads to the programmer, i.e. a
well understood concept.

> But Guido and others seem to be
> objecting to my implicit-cocall proposal on the basis that
> it looks *too much* like normal code. It seems to me that
> the same criticism should apply even more to Stackless.

I think an important part of the criticism is
hiding the fact that you are writing a cofunction
away inside the function definition itself.

Generators have the same problem, but at least you
can call them as regular Python functions and they
only work a little different than normal functions.

>> The techniques used by Stackless to achieve this are nasty,
>> but then
>> Python also ships with ctypes which relies on similar nasty techniques
> But at least it works provided you write your ctypes code
> correctly and the library you're calling isn't buggy. I
> seem to remember that there are certain C libraries that
> break Stackless because they assume that their C stack
> frames don't move around.

Well, viruses will have a harder time for sure ;-) I am not
aware of other use cases that would need to know the
location of the stack frame in memory.

BTW: I'm sure that the functionality needed by Stackless could
also be moved into a C lib for other languages to use
(much like the libffi code).

Marc-Andre Lemburg

Professional Python Services directly from the Source  (#1, Aug 16 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611

More information about the Python-ideas mailing list