[Python-ideas] Thoughts on the future of concurrency in Python: PEP 3148

[Steven Hazel]

On Sunday, March 17, 2013 2:06:48 PM UTC-7, Guido van Rossum wrote:
>
> As Nick pointed out, this water is already under the 3.2 bridge. Too 
> late to change now. 
>

Oops, didn't realize it was already out. In that case, please take this as 
a suggestion that the docs might benefit from an explanation like "A Future 
is basically just a callback registry. Rather than taking callbacks as 
parameters, functions can return a Future, where callbacks can be 
registered with Future.add_done_callback."
 

> I actually want to de-emphasize the fact that Futures hold callbacks. 
> I want people to think them as magic "wait points" that combine with 
> "yield from" to return a pseudo-synchronous result. In fact, I want to 
> altogether de-emphasize that async I/O is done under the hood using 
> callbacks. Only implementers of async frameworks should need to know 
> that, for the most part. 
>

It's true that when you're using a generator-based async framework, you can 
and should think of a Future as a "thing you can wait on" most of the time. 
My experience with monocle though is that it is helpful rather than harmful 
to reveal that they're about callbacks. In the early days of monocle, we 
were using Deferreds, which a lot of new monocle users didn't really 
understand, and people tended to get very confused about what kinds of 
things a Deferred could possibly be doing. Explaining things in terms of 
callbacks was helpful in getting people to understand monocle.

I don't think you can really abstract the idea of callbacks away from 
Futures without making them more mystifying. Callbacks are not an 
implementation detail of Futures, they're essential to what Futures do, so 
essential that I think CallbackRegistry is pretty good alternative name, 
and in monocle we actually called the class Callback. I sometimes explain 
that a monocle oroutine is "yielding on a callback" — meaning it's waiting 
now and it'll resume when called back. The name helps explain what's 
happening, even for an user of the framework.

Hm. I find the idea of callbacks far from simple nor familiar. 
>

Many programmers have used callbacks in other contexts. But, even when they 
haven't, "you pass in a function that will get called when the operation is 
done" is an idea I've watched many people grasp immediately the first time 
they saw it. In contrast, Futures, which are almost the same idea, are 
often viewed as sophisticated black magic, and I've more than once heard 
them explained in terms of *time travel*.
 

> It's a threadpool. 
>

Oh, I see. Well, for what it's worth, including this next to Futures 
confused me. I kind of implied to me when I read the PEP that Executors are 
somehow necessary to using Futures, when in fact they're just one of many 
contexts in which a Future might be a good API.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130317/f5725032/attachment.html>