[Python-ideas] Cofunctions - Back to Basics

[Greg Ewing]

Steven D'Aprano wrote:

> (1) You state that it is a "special kind of generator", but don't give 
> any clue as to how it is special.

You're supposed to read the rest of the specification to find that
out.

> (2) Cofunctions, apparently, "may" contain yield or yield from. 
> Presumably that means that yield is optional, otherwise it would be 
> "must" rather than "may". So what sort of generator do you get without a 
> yield? The PEP doesn't give me any clue.

A cofunction needs to be able to 'yield', because that's the
way you suspend a cofunction-based coroutine. It needs to still
be a generator even if it doesn't contain a 'yield', because
it may call another cofunction that does yield. So the fact
that it's defined with 'codef' makes it a generator, regardless
of whether it directly contains any yields.

> "From the outside, the distinguishing feature of a cofunction is that it
> cannot be called directly from within the body of an ordinary function. 
> An exception is raised if such a call to a cofunction is attempted."
> 
> Many things can't be called as functions: ints, strings, lists, etc. It 
> simply isn't clear to me how cofunctions are different from any other 
> non-callable object.

I mean it's what distinguishes cofunctions from functions defined
with 'def'. The comparison is between cofunctions and normal functions,
not between cofunctions and any non-callable object.

> the above as stated implies the following:
> 
> def spam():
>     x = cofunction()  # fails, since directly inside a function
> 
> x = cofunction()  # succeeds, since not inside a function
> 
> Surely that isn't what you mean to imply, is it?

You're right, I didn't mean to imply that. A better way to phrase it
would be "A cofunction can only be called directly from the body of
another cofunction. An exception is raised if an attempt is made
to call it in any other context."

> Is there any prior art in other languages? I have googled on 
> "cofunction", and I get many, many hits to the concept from mathematics 
> (e.g. sine and cosine) but virtually nothing in programming circles.

It's my own term, not based on any prior art that I'm aware of.
It seemed like a natural way to combine the notion of "coroutine"
with "function" in the Python sense of something defined with 'def'.

(I have seen the word used once in a paper relating to functional
programming. The author drew a distinction between "functions"
operating on finite data, and "cofunctions" operating on "cofinite"
"codata". But apart from the idea of dividing functions into two
disjoint classes, it was unrelated to what I'm talking about.)

> In the Motivation and Rationale section, you state:
> 
>     If one forgets to use ``yield from`` when it should have
>     been used, or uses it when it shouldn't have, the symptoms
>     that result can be extremely obscure and confusing.
> 
> I don't believe that remembering to write ``codef`` instead of ``def`` 
> is any easier than remembering to write ``yield from`` instead of 
> ``yield`` or ``return``.

It's easier because if you forget, you get told about it loudly
and clearly.

Whereas if you forget to write "yield from" where you should have,
nothing goes wrong immediately -- the call succeeds and returns
something. The trouble is, it's an iterator rather than the function
return value you were expecting. If you're lucky, this will trip
up something not too much further down. If you're unlucky, the
incorrect value will get returned to a higher level or stored
away somewhere, to cause problems much later when it's far from
obvious where it came from.

The reason this is an issue is that it's much easier to make this
kind of mistake when using generators as coroutines rather than
iterators. Normally when you call a generator, the purpose you
have in mind is "produce a stream of things", in which case it's
obvious that you need to do something more than just a call,
such as iterating over the result or using "yield from" to pass
them on to your caller.

But when using yield-from to call a subfunction in a coroutine,
the purpose you have in mind is not "produce a stream of things"
but simply "do something" or "calculate a value". And this is
the same as the purpose you have in mind for all the ordinary calls
that *don't* require -- and in fact must *not* have -- yield-from
in front of them. So there is great room for confusion!

What's more, when using generators in the usual way, the thing
you're calling is designed from the outset to return an iterator
as part of its contract, and that is not likely to change.

However, it's very likely that parts of your coroutine that
initially were just ordinary functions will later need to be
able to suspend themselves. When that happens, you need to
track down all the places you call it from and add "yield from"
in front of them -- all the time wrestling with the kinds of
less-than-obvious symptoms that I described above.

With cofunctions, on the other hand, this process is straightforward
and almost automatic. The interpreter will tell you exactly
which functions have a problem, and you fix them by changing
their definitions from 'def' to 'codef'.

-- 
Greg