[Python-ideas] PEP draft - Composable futures for reactive programming

[Guido van Rossum]

On Sat, Dec 21, 2013 at 2:48 PM, Sergii Mikhtoniuk <mikhtonyuk at gmail.com> wrote:
> Hello Guido, Nick,
>
> Thanks a lot for your valuable feedback.

You're welcome.

> Indeed there is a lot of overlap between asyncio and concurrent.futures
> packages, so it would be very interesting to hear your overall thoughts on
> role/future of concurrent package. Do you consider it rudimentary and
> replaceable by asyncio.Future completely?

They don't really compare. concurrent.futures is about *threads*.
asyncio.Future is about *avoiding* threads in favor of more
lightweight "tasks" and "coroutines", which in turn are built on top
of lower-level callbacks.

(While I want to get away from callbacks as a programming paradigm,
asyncio uses them at the lower levels both because they are a logical
low-level building block and for interoperability with other
frameworks like Tornado and Twisted.)

> I think the question is even not much about which is your preferred
> implementation, but rather do we want having futures as stand-alone package
> or not. Do you see all these implementations converging in future?

I see them as not even competing. They use different paradigms and
apply to different use cases.

> To me Future is a very simple and self-contained primitive, independent of
> thread pools, processes, IO, and networking.

(Agreed on the I/O and networking part only.)

> One thing concurrent.futures
> package does a very good job at is defining an isolated namespace for
> futures, stressing out this clear boundary (let’s disregard here
> ThreadPoolExecutor and ProcessPoolExecutor classes which for some reason
> ended up in it too).

Actually the executor API is an important and integral part of that
package, and threads underlie everything you can do with its Futures.

> So when I think of schedulers and event loops implementations I see them as
> ones that build on top of the Future primitive, not providing it as part of
> their implementation. What I think is important is that having unified
> Future class simplifies interoperability between different kinds of
> schedulers, not necessarily associated with asyncio event loops (process
> pools for example).

The interoperability is completely missing. I can't tell if you've
used asyncio at all, but the important operation of *waiting* for a
result is fundamentally different there than in concurrent.futures. In
the latter, you just write "x = f.result()" and your thread blocks
until the result is available. In asyncio, you have to write "x =
yield from f.result()" which is a coroutine block that lets other
tasks run in the same thread. ("yield from" in this case is how Python
spells the operation that C# calls "await").

> Futures are definitely not the final solution for concurrency problem but
> rather a well-established utility for representing async operations. It is a
> job of higher layer systems to provide more convenient ways for dealing with
> asyncs (yield from coroutines, async/await rewrites etc.),

But unless you are proposing some kind of radical change to add
compile-time type checking/inference to Python, the rewrite option is
unavailable in Python.

> so I would not
> say that futures encourage callback-style programming,

The concurrent.futures.Future class does not. But unless I misread
your proposal, your extensions do.

> it’s simply a
> lower-layer functionality. On the contrary, monadic

(Say that word one more time and everyone tunes out. :-)

> methods for futures
> composition (e.g. map(), all(), first() etc.) ensure that no errors would be
> lost in the process, so I think they would complement yield from model quite
> nicely by hiding complexity of state maintenance from user and reducing the
> number of back-and-forth communications between event loop and coroutines.

I'm not sure I follow. Again, I'm not sure if you've actually written
any code using asyncio.

TBH I've written a fair number of example programs for asyncio and
I've very rarely felt the need for these composition functions. The
main composition primitive I tend to use is "yield from".

> Besides Futures, reactive programming

What *is* reactive programming? If you're talking about
http://en.wikipedia.org/wiki/Reactive_programming,
I'm not sure that it maps well to Python.

> has more utilities to offer, such as
> Observables (representing asynchronous streams of values). It is also a very
> useful abstraction with a rich set of composition strategies (merging,
> concatenation, grouping), and may deserve its place in separate package.

It all sounds very abstract and academic. :-)

> Hope to hear back from you to get better picture on overall design direction
> here before jumping to implementation details.
>
> Brief follow-up to your questions:
>  - Idea behind the Future/Promise separation is to draw a clean line between
> client-facing and scheduler-side APIs respectively. Futures should not
> expose any completion methods, which clients should not call anyway.

Given that Future and Promise are often used synonymously, using them
to make this distinction sounds confusing. I agree that Futures have
two different APIs, one for the consumer and another for the producer.
But I'm not sure that it's necessary to separate them more strictly --
convention seems good enough to me here. (It's the same with many
communication primitives, like queues and even threads.)

(The one thing that trips people up frequently is that, while
set_result() and set_exception() are producer APIs, cancel() is a
consumer API, and the cancellation signal travels from the consumer to
the producer.)

>  - Completely agree with you that Twisted-style callbacks are evil and it is
> better to have single code path for getting result or raising exception
>  - Sorry for bad grammar in the proposal, it’s an early draft written in 3
> AM, so I will definitely improve on it if we decide to move forward.

No problem!

> Thanks,
> Sergii

-- 
--Guido van Rossum (python.org/~guido)