[Python-ideas] Thoughts on the future of concurrency in Python: PEP 3148

[Guido van Rossum]

Hi Steve!

(Whoops, pressed the wrong button.)

On Sun, Mar 17, 2013 at 11:34 AM, Steven Hazel <sah at awesame.org> wrote:
> Hi,
>
> I'm one of the authors of monocle (https://github.com/saucelabs/monocle), an
> async Python programming framework with a blocking look-alike syntax based
> on generators. We've been using monocle at my startup Sauce Labs, and I used
> its predecessor on a small team at BitTorrent. I think it's met with some
> amazing success vs. similar projects using threads. In about five years of
> working with teams with some members who haven't always understood monocle
> very well, we've never written a deadlock, for example.

I'm really sorry -- in my keynote I forgot to mention that Monocle was
one of my inspirations for using yield to do async things in NDB,
which I did for App Engine. Monocole has greatly guided my thoughts
(more than Twisted's somewhat-equivalent inlineCallbacks).

> I've been keeping up with PEPs and skimming python-ideas discussions about
> concurrency-related features in Python, and feeling unsure about how best to
> jump into the conversation. Today it occurred to me that sooner is better
> and I should just get all my various opinions out there in case they're
> helpful.

Great!

> First of all, I want to reiterate: while I have mainly criticism to offer,
> it is only intended to be helpful. I don't have a lot of time to devote to
> this myself, so I'm trying to do what I can to be useful to those who do.
>
> I think I'll take this one PEP at a time, and start with the easiest one.
>
> PEP 3148 - Futures
>
> First, I like the idea of this PEP. I think it'll improve interoperability
> between async frameworks in Python.
>
> I believe the name "future" is a poor choice. After many years of explaining
> this concept to programmers, I've found that its various abstract-conceptual
> names do a lot to confuse people. Twisted's "deferred" is probably the worst
> offender here, but both "future" and "promise" also make this very simple
> construct sound complicated, and it's a crime.

As Nick pointed out, this water is already under the 3.2 bridge. Too
late to change now.

> There are two ways I've found to explain this that seem to keep people from
> getting confused:
>
> The first is, call it a "callback registry". Many programmers are familiar
> with the idea of passing callbacks into a function. You just explain that
> here, instead of taking the callbacks as parameters, we return a callback
> registry object and you can add your callbacks there. (If the operation is
> already done they'll just get called when you add them.)

I actually want to de-emphasize the fact that Futures hold callbacks.
I want people to think them as magic "wait points" that combine with
"yield from" to return a pseudo-synchronous result. In fact, I want to
altogether de-emphasize that async I/O is done under the hood using
callbacks. Only implementers of async frameworks should need to know
that, for the most part.

> The second is a variant on that: say that this function, instead of taking a
> callback as a parameter, returns its callback. That sounds useless, but
> fortunately the callback it returns is a callable object, and it also has an
> "add" method that lets you add functions that it will call when called back.
> So it's a callback that you can manipulate to do what you want. (This is
> what we do in monocle:
> https://github.com/saucelabs/monocle/blob/master/monocle/callback.py)
>
> These approaches, by focusing on the simple and familiar idea of callbacks,
> do a lot to eliminate confusing ideas about communicating with the future of
> the process.

Hm. I find the idea of callbacks far from simple nor familiar.

> Finally, I'm not sure I get the point of the Executor class in PEP 3148. It
> seems to me that any real implementation of that class needs a call_later
> method for scheduling operations, and a way of connecting up to event-based
> IO APIs. I don't really understand what Executor would be good for.

It's a threadpool.

> Thanks for reading.

Thanks for thinking with us. And thanks for Monocle! (And for paying
to fly Raymond to EuroPython a few years ago so I could hear him talk
about it. :-)

-- 
--Guido van Rossum (python.org/~guido)