[Python-ideas] PEP draft - Composable futures for reactive programming

[Guido van Rossum]

There's still the issue that in the threading version, you wait for a
Future by blocking the current thread, while in the asyncio version,
you must use "yield from" to block. For interoperability you would
have to refrain from *any* blocking operations (including "yield
from") so you would only be able to use callbacks. But whether you had
to write "x = f.result()" or "x = concurrent.futures.wait_for(f)",
either way you'd implicitly be blocking the current thread.

Yes, a clever scheduler could run other callbacks while blocking, but
that's not a complete solution, because another callback might do a
similar blocking operation, and whatever that waited for could hold up
the earlier blocking operation, causing an ever-deeper recursion and
of event loop invocations that might never complete. (I've had to
debug this in production code.) To cut through that you'd have to have
some kind of stack-swapping coroutine implementation like gevent, or a
syntactic preprocessor that inserts yield or yield-from operations
(I've heard from people who do this), or you'd need a clairvoyant
scheduler that would know which callbacks won't block.

I like the C# solution, but it depends on static typing so a compiler
can know when to emit the coroutine interactions. That wouldn't work
in Python, unless you made the compiler recognizing the wait_for()
operation by name, which feels unsavory (although we do it for super()
:-).

I guess for extreme interop, callbacks that never block is your only
option anyway, but I'd be sad if we had to to recommend this as the
preferred paradigm, or claim that it is all you need.

--Guido (if I don't respond to this thread for the next two weeks,
it's because I'm on vacation :-)


On Sat, Dec 21, 2013 at 5:37 PM, Ben Darnell <ben at bendarnell.com> wrote:
> On Sat, Dec 21, 2013 at 7:26 PM, Guido van Rossum <guido at python.org> wrote:
>>
>> On Sat, Dec 21, 2013 at 2:48 PM, Sergii Mikhtoniuk <mikhtonyuk at gmail.com>
>> wrote:
>> > Indeed there is a lot of overlap between asyncio and concurrent.futures
>> > packages, so it would be very interesting to hear your overall thoughts
>> > on
>> > role/future of concurrent package. Do you consider it rudimentary and
>> > replaceable by asyncio.Future completely?
>>
>> They don't really compare. concurrent.futures is about *threads*.
>> asyncio.Future is about *avoiding* threads in favor of more
>> lightweight "tasks" and "coroutines", which in turn are built on top
>> of lower-level callbacks.
>
>
> concurrent.futures.ThreadPoolExecutor is about threads; the Future class
> itself is broader.  When I integrated Futures into Tornado I used
> concurrent.futures.Future directly (when available).  asyncio.Future is just
> an optimized version of c.f.Future (the optimization comes from assuming
> single-threaded usage).  There should at least be a common ABC between them.
>
>>
>>
>> (While I want to get away from callbacks as a programming paradigm,
>> asyncio uses them at the lower levels both because they are a logical
>> low-level building block and for interoperability with other
>> frameworks like Tornado and Twisted.)
>>
>> > I think the question is even not much about which is your preferred
>> > implementation, but rather do we want having futures as stand-alone
>> > package
>> > or not. Do you see all these implementations converging in future?
>>
>> I see them as not even competing. They use different paradigms and
>> apply to different use cases.
>>
>> > To me Future is a very simple and self-contained primitive, independent
>> > of
>> > thread pools, processes, IO, and networking.
>>
>> (Agreed on the I/O and networking part only.)
>>
>> > One thing concurrent.futures
>> > package does a very good job at is defining an isolated namespace for
>> > futures, stressing out this clear boundary (let’s disregard here
>> > ThreadPoolExecutor and ProcessPoolExecutor classes which for some reason
>> > ended up in it too).
>>
>> Actually the executor API is an important and integral part of that
>> package, and threads underlie everything you can do with its Futures.
>>
>> > So when I think of schedulers and event loops implementations I see them
>> > as
>> > ones that build on top of the Future primitive, not providing it as part
>> > of
>> > their implementation. What I think is important is that having unified
>> > Future class simplifies interoperability between different kinds of
>> > schedulers, not necessarily associated with asyncio event loops (process
>> > pools for example).
>>
>> The interoperability is completely missing. I can't tell if you've
>> used asyncio at all, but the important operation of *waiting* for a
>> result is fundamentally different there than in concurrent.futures. In
>> the latter, you just write "x = f.result()" and your thread blocks
>> until the result is available. In asyncio, you have to write "x =
>> yield from f.result()" which is a coroutine block that lets other
>> tasks run in the same thread. ("yield from" in this case is how Python
>> spells the operation that C# calls "await").
>
>
> The way I see it, the fundamental operation on Futures is add_done_callback.
> We then have various higher-level operations that let us get away from using
> callbacks directly.  One of these happens to be a method on Future: the
> blocking mode of Future.result().  Another is implemented in asyncio and
> Tornado, in the ability to "yield" a Future.  The blocking mode of result()
> is just a convenience; if asyncio-style futures had existed first then we
> could instead have a function like "x = concurrent.futures.wait_for(f)".  In
> fact, you could write this wait_for function today in a way that works for
> both concurrent and asyncio futures.
>
> This is already interoperable:  Tornado's Resolver interface
> (https://github.com/facebook/tornado/blob/master/tornado/netutil.py#L184)
> returns a Future, which may be generated by a ThreadPoolExecutor or an
> asynchronous wrapper around pycares or twisted.  In the other direction I've
> worked on hybrid apps that have one Tornado thread alongside a bunch of
> Django threads; in these apps it would work to have a Django thread block on
> f.result() for a Future returned by Tornado's AsyncHTTPClient.
>
> -Ben
>
>>
>> > Futures are definitely not the final solution for concurrency problem
>> > but
>> > rather a well-established utility for representing async operations. It
>> > is a
>> > job of higher layer systems to provide more convenient ways for dealing
>> > with
>> > asyncs (yield from coroutines, async/await rewrites etc.),
>>
>> But unless you are proposing some kind of radical change to add
>> compile-time type checking/inference to Python, the rewrite option is
>> unavailable in Python.
>>
>> > so I would not
>> > say that futures encourage callback-style programming,
>>
>> The concurrent.futures.Future class does not. But unless I misread
>> your proposal, your extensions do.
>>
>> > it’s simply a
>> > lower-layer functionality. On the contrary, monadic
>>
>> (Say that word one more time and everyone tunes out. :-)
>>
>> > methods for futures
>> > composition (e.g. map(), all(), first() etc.) ensure that no errors
>> > would be
>> > lost in the process, so I think they would complement yield from model
>> > quite
>> > nicely by hiding complexity of state maintenance from user and reducing
>> > the
>> > number of back-and-forth communications between event loop and
>> > coroutines.
>>
>> I'm not sure I follow. Again, I'm not sure if you've actually written
>> any code using asyncio.
>>
>> TBH I've written a fair number of example programs for asyncio and
>> I've very rarely felt the need for these composition functions. The
>> main composition primitive I tend to use is "yield from".
>>
>> > Besides Futures, reactive programming
>>
>> What *is* reactive programming? If you're talking about
>> http://en.wikipedia.org/wiki/Reactive_programming,
>> I'm not sure that it maps well to Python.
>>
>> > has more utilities to offer, such as
>> > Observables (representing asynchronous streams of values). It is also a
>> > very
>> > useful abstraction with a rich set of composition strategies (merging,
>> > concatenation, grouping), and may deserve its place in separate package.
>>
>> It all sounds very abstract and academic. :-)
>>
>> > Hope to hear back from you to get better picture on overall design
>> > direction
>> > here before jumping to implementation details.
>> >
>> > Brief follow-up to your questions:
>> >  - Idea behind the Future/Promise separation is to draw a clean line
>> > between
>> > client-facing and scheduler-side APIs respectively. Futures should not
>> > expose any completion methods, which clients should not call anyway.
>>
>> Given that Future and Promise are often used synonymously, using them
>> to make this distinction sounds confusing. I agree that Futures have
>> two different APIs, one for the consumer and another for the producer.
>> But I'm not sure that it's necessary to separate them more strictly --
>> convention seems good enough to me here. (It's the same with many
>> communication primitives, like queues and even threads.)
>>
>> (The one thing that trips people up frequently is that, while
>> set_result() and set_exception() are producer APIs, cancel() is a
>> consumer API, and the cancellation signal travels from the consumer to
>> the producer.)
>>
>> >  - Completely agree with you that Twisted-style callbacks are evil and
>> > it is
>> > better to have single code path for getting result or raising exception
>> >  - Sorry for bad grammar in the proposal, it’s an early draft written in
>> > 3
>> > AM, so I will definitely improve on it if we decide to move forward.
>>
>> No problem!
>>
>> > Thanks,
>> > Sergii
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
>



-- 
--Guido van Rossum (python.org/~guido)