[Python-ideas] Cofunctions PEP - Revision 4

Thu Aug 12 23:15:31 CEST 2010

Apologies if this already exists, but for the benefit of those less enlightened, I think it would be very helpful if the pep included or linked to an example of an algorithm implemented 3 ways:

- Straight python, no coroutines
- Coroutines implemented via "yield from"
- Coroutines implemented via "cocall"

iirc, the last two would not look much different, but maybe I'm mistaken.

As I understand it:

cocall f(x, y, z)

is sugar for:

yield from f.__cocall__(x, y, z)

and it now magically promotes the function that contains it to a cofunction (thus implementing __cocall__ for said function).

From what I understand, __cocall__ does not exist because you might want to also have __call__ with different behavior, but instead it exists to allow the "cocaller" to differentiate between cofunctions and normal functions? In theory though, I could implement an object myself that implemented both __call__ and __cocall__, correct? I suppose __cocall__ is to __call__ as __iter__ is to __call__ presently.

I'd say my main problem with this is conceptual complexity. Many folks have a hard time understanding generators, and the presence of the conceptually similar language concept of iterators doesn't help. This feels like yet another conceptually similar concept to generators, that just muddies the waters further.

What I'd love to see is a version of coroutines that isn't just sugared generators. It really seems to me that generators should be implemented on top of coroutines and the not the reverse. That would lead to a more linear path to understanding: iterators -> generators -> coroutines. This proposal doesn't feel like that to me, it feels more like an adjunct thing that uses the generator machinery for different ends.

I'm surely confused, but that's part of my point 8^)

-Casey

On Aug 11, 2010, at 2:03 AM, Greg Ewing wrote:

> Here's an updated version of the PEP reflecting my
> recent suggestions on how to eliminate 'codef'.
> 
> PEP: XXX
> Title: Cofunctions
> Version: $Revision$
> Last-Modified: $Date$
> Author: Gregory Ewing <greg.ewing at canterbury.ac.nz>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 13-Feb-2009
> Python-Version: 3.x
> Post-History:
> 
> 
> Abstract
> ========
> 
> A syntax is proposed for defining and calling a special type of generator
> called a 'cofunction'.  It is designed to provide a streamlined way of
> writing generator-based coroutines, and allow the early detection of
> certain kinds of error that are easily made when writing such code, which
> otherwise tend to cause hard-to-diagnose symptoms.
> 
> This proposal builds on the 'yield from' mechanism described in PEP 380,
> and describes some of the semantics of cofunctions in terms of it. However,
> it would be possible to define and implement cofunctions independently of
> PEP 380 if so desired.
> 
> 
> Specification
> =============
> 
> Cofunction definitions
> ----------------------
> 
> A cofunction is a special kind of generator, distinguished by the presence
> of the keyword ``cocall`` (defined below) at least once in its body. It may
> also contain ``yield`` and/or ``yield from`` expressions, which behave as
> they do in other generators.
> 
> From the outside, the distinguishing feature of a cofunction is that it cannot
> be called the same way as an ordinary function. An exception is raised if an
> ordinary call to a cofunction is attempted.
> 
> Cocalls
> -------
> 
> Calls from one cofunction to another are made by marking the call with
> a new keyword ``cocall``. The expression
> 
> ::
> 
>    cocall f(*args, **kwds)
> 
> is evaluated by first checking whether the object ``f`` implements
> a ``__cocall__`` method. If it does, the cocall expression is
> equivalent to
> 
> ::
> 
>    yield from f.__cocall__(*args, **kwds)
> 
> except that the object returned by __cocall__ is expected to be an
> iterator, so the step of calling iter() on it is skipped.
> 
> If ``f`` does not have a ``__cocall__`` method, or the ``__cocall__``
> method returns ``NotImplemented``, then the cocall expression is
> treated as an ordinary call, and the ``__call__`` method of ``f``
> is invoked.
> 
> Objects which implement __cocall__ are expected to return an object
> obeying the iterator protocol. Cofunctions respond to __cocall__ the
> same way as ordinary generator functions respond to __call__, i.e. by
> returning a generator-iterator.
> 
> Certain objects that wrap other callable objects, notably bound methods,
> will be given __cocall__ implementations that delegate to the underlying
> object.
> 
> Grammar
> -------
> 
> The full syntax of a cocall expression is described by the following
> grammar lines:
> 
> ::
> 
>    atom: cocall | <existing alternatives for atom>
>    cocall: 'cocall' atom cotrailer* '(' [arglist] ')'
>    cotrailer: '[' subscriptlist ']' | '.' NAME
> 
> Note that this syntax allows cocalls to methods and elements of sequences
> or mappings to be expressed naturally. For example, the following are valid:
> 
> ::
> 
>    y = cocall self.foo(x)
>    y = cocall funcdict[key](x)
>    y = cocall a.b.c[i].d(x)
> 
> Also note that the final calling parentheses are mandatory, so that for example
> the following is invalid syntax:
> 
> ::
> 
>    y = cocall f     # INVALID
> 
> New builtins, attributes and C API functions
> --------------------------------------------
> 
> To facilitate interfacing cofunctions with non-coroutine code, there will
> be a built-in function ``costart`` whose definition is equivalent to
> 
> ::
> 
>    def costart(obj, *args, **kwds):
>        try:
>            m = obj.__cocall__
>        except AttributeError:
>            result = NotImplemented
>        else:
>            result = m(*args, **kwds)
>        if result is NotImplemented:
>            raise TypeError("Object does not support cocall")
>        return result
> 
> There will also be a corresponding C API function
> 
> ::
> 
>    PyObject *PyObject_CoCall(PyObject *obj, PyObject *args, PyObject *kwds)
> 
> It is left unspecified for now whether a cofunction is a distinct type
> of object or, like a generator function, is simply a specially-marked
> function instance. If the latter, a read-only boolean attribute
> ``__iscofunction__`` should be provided to allow testing whether a given
> function object is a cofunction.
> 
> 
> Motivation and Rationale
> ========================
> 
> The ``yield from`` syntax is reasonably self-explanatory when used for the
> purpose of delegating part of the work of a generator to another function. It
> can also be used to good effect in the implementation of generator-based
> coroutines, but it reads somewhat awkwardly when used for that purpose, and
> tends to obscure the true intent of the code.
> 
> Furthermore, using generators as coroutines is somewhat error-prone. If one
> forgets to use ``yield from`` when it should have been used, or uses it when it
> shouldn't have, the symptoms that result can be extremely obscure and confusing.
> 
> Finally, sometimes there is a need for a function to be a coroutine even though
> it does not yield anything, and in these cases it is necessary to resort to
> kludges such as ``if 0: yield`` to force it to be a generator.
> 
> The ``cocall`` construct address the first issue by making the syntax directly
> reflect the intent, that is, that the function being called forms part of a
> coroutine.
> 
> The second issue is addressed by making it impossible to mix coroutine and
> non-coroutine code in ways that don't make sense. If the rules are violated, an
> exception is raised that points out exactly what and where the problem is.
> 
> Lastly, the need for dummy yields is eliminated by making it possible for a
> cofunction to call both cofunctions and ordinary functions with the same syntax,
> so that an ordinary function can be used in place of a cofunction that yields
> zero times.
> 
> 
> Record of Discussion
> ====================
> 
> An earlier version of this proposal required a special keyword ``codef`` to be
> used in place of ``def`` when defining a cofunction, and disallowed calling an
> ordinary function using ``cocall``.  However, it became evident that these
> features were not necessary, and the ``codef`` keyword was dropped in the
> interests of minimising the number of new keywords required.
> 
> The use of a decorator instead of ``codef`` was also suggested, but the current
> proposal makes this unnecessary as well.
> 
> It has been questioned whether some combination of decorators and functions
> could be used instead of a dedicated ``cocall`` syntax. While this might be
> possible, to achieve equivalent error-detecting power it would be necessary
> to write cofunction calls as something like
> 
> ::
> 
>    yield from cocall(f)(args)
> 
> making them even more verbose and inelegant than an unadorned ``yield from``.
> It is also not clear whether it is possible to achieve all of the benefits of
> the cocall syntax using this kind of approach.
> 
> 
> Prototype Implementation
> ========================
> 
> An implementation of an earlier version of this proposal in the form of patches
> to Python 3.1.2 can be found here:
> 
> http://www.cosc.canterbury.ac.nz/greg.ewing/python/generators/cofunctions.html
> 
> If this version of the proposal is received favourably, the implementation will
> be updated to match.
> 
> 
> Copyright
> =========
> 
> This document has been placed in the public domain.
> 
> 
> 
> ..
>   Local Variables:
>   mode: indented-text
>   indent-tabs-mode: nil
>   sentence-end-double-space: t
>   fill-column: 70
>   coding: utf-8
>   End:
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas