[Python-ideas] Thoughts on the future of concurrency in Python: PEP 3148

[Guido van Rossum]

On Sun, Mar 17, 2013 at 3:14 PM, Steven Hazel <sah at awesame.org> wrote:
> On Sunday, March 17, 2013 2:06:48 PM UTC-7, Guido van Rossum wrote:
>> As Nick pointed out, this water is already under the 3.2 bridge. Too
>> late to change now.
>
> Oops, didn't realize it was already out. In that case, please take this as a
> suggestion that the docs might benefit from an explanation like "A Future is
> basically just a callback registry. Rather than taking callbacks as
> parameters, functions can return a Future, where callbacks can be registered
> with Future.add_done_callback."

But that's not how I want users to think about Futures at all
(certainly not in the async I/O world). You should almost never have
to think about the callbacks registered with the Future! (Unless you
are implementing an adapter for a callback-based framework -- but
that's presumably a fairly uncommon activity.)

>> I actually want to de-emphasize the fact that Futures hold callbacks.
>> I want people to think them as magic "wait points" that combine with
>> "yield from" to return a pseudo-synchronous result. In fact, I want to
>> altogether de-emphasize that async I/O is done under the hood using
>> callbacks. Only implementers of async frameworks should need to know
>> that, for the most part.
>
> It's true that when you're using a generator-based async framework, you can
> and should think of a Future as a "thing you can wait on" most of the time.
> My experience with monocle though is that it is helpful rather than harmful
> to reveal that they're about callbacks. In the early days of monocle, we
> were using Deferreds, which a lot of new monocle users didn't really
> understand, and people tended to get very confused about what kinds of
> things a Deferred could possibly be doing. Explaining things in terms of
> callbacks was helpful in getting people to understand monocle.

I'm guessing that there are two quite distinct audiences for the docs:
those new to async programming, and those who have used a
callback-based framework before. For the latter, it certainly makes
sense to clarify how Futures work, and I do not intend to hide it -- I
just don't think this is the primary audience for the docs. If you've
used callbacks before, you can find out how Futures work in the
reference docs.

> I don't think you can really abstract the idea of callbacks away from
> Futures without making them more mystifying. Callbacks are not an
> implementation detail of Futures, they're essential to what Futures do, so
> essential that I think CallbackRegistry is pretty good alternative name, and
> in monocle we actually called the class Callback. I sometimes explain that a
> monocle oroutine is "yielding on a callback" — meaning it's waiting now and
> it'll resume when called back. The name helps explain what's happening, even
> for an user of the framework.

I respectfully disagree. I am currently writing and/or reviewing reams
of code that *uses* Futures to implement complex protocols (currently
we're working on an HTTP client and server). This code uses yield-from
all over the place, and not once is there a need to use a callback.

>> Hm. I find the idea of callbacks far from simple nor familiar.
>
> Many programmers have used callbacks in other contexts. But, even when they
> haven't, "you pass in a function that will get called when the operation is
> done" is an idea I've watched many people grasp immediately the first time
> they saw it.

Easy to grasp, but leads to lots of trouble later on. I'd say that the
concept of "GO TO" is equally graspable -- but that doesn't make it a
good concept to start with.

> In contrast, Futures, which are almost the same idea, are often
> viewed as sophisticated black magic, and I've more than once heard them
> explained in terms of *time travel*.

Sigh. I will make sure that won't happen in any of the docs that *I*
write. The concept is more closely related to the financial concept
with the same name. I suppose I could call them Promises instead, but
Future seem to have caught on already (PEP 3148, and it's also what
Java uses). Also note that explicit Futures are a lot less magical
than implicit Futures.

Finally, I have not had any issues explaining the use of Futures in
NDB, where they are being used by (many) thousands of users with only
an average grasp of Python concepts.

>> It's a threadpool.
>
> Oh, I see. Well, for what it's worth, including this next to Futures
> confused me. I kind of implied to me when I read the PEP that Executors are
> somehow necessary to using Futures, when in fact they're just one of many
> contexts in which a Future might be a good API.

Take it up with the authors of PEP 3148, or submit a doc bug for the
stdlib docs explaining them:
http://docs.python.org/3/library/concurrent.futures.html?highlight=executor#concurrent.futures.Executor

--
--Guido van Rossum (python.org/~guido)