[Python-ideas] PEP draft - Composable futures for reactive programming

[Guido van Rossum]

Aha. That is clever. I will have to look into the details more, but the
idea is promising. Sorry I didn't see tty his before.
On Dec 22, 2013 12:30 PM, "Ben Darnell" <ben at bendarnell.com> wrote:

> On Sat, Dec 21, 2013 at 10:45 PM, Guido van Rossum <guido at python.org>wrote:
>
>> There's still the issue that in the threading version, you wait for a
>> Future by blocking the current thread, while in the asyncio version,
>> you must use "yield from" to block. For interoperability you would
>> have to refrain from *any* blocking operations (including "yield
>> from") so you would only be able to use callbacks. But whether you had
>> to write "x = f.result()" or "x = concurrent.futures.wait_for(f)",
>> either way you'd implicitly be blocking the current thread.
>>
>
> Threaded *consumers* of Futures wait for them by blocking, while
> asynchronous consumers wait for them by yielding.  It doesn't matter
> whether the *producer* of the Future is threaded or asynchronous (except
> that if you know you won't be using threads you can use a faster
> thread-unsafe Future implementation).
>
> -Ben
>
>
>>
>> Yes, a clever scheduler could run other callbacks while blocking, but
>> that's not a complete solution, because another callback might do a
>> similar blocking operation, and whatever that waited for could hold up
>> the earlier blocking operation, causing an ever-deeper recursion and
>> of event loop invocations that might never complete. (I've had to
>> debug this in production code.) To cut through that you'd have to have
>> some kind of stack-swapping coroutine implementation like gevent, or a
>> syntactic preprocessor that inserts yield or yield-from operations
>> (I've heard from people who do this), or you'd need a clairvoyant
>> scheduler that would know which callbacks won't block.
>>
>> I like the C# solution, but it depends on static typing so a compiler
>> can know when to emit the coroutine interactions. That wouldn't work
>> in Python, unless you made the compiler recognizing the wait_for()
>> operation by name, which feels unsavory (although we do it for super()
>> :-).
>>
>> I guess for extreme interop, callbacks that never block is your only
>> option anyway, but I'd be sad if we had to to recommend this as the
>> preferred paradigm, or claim that it is all you need.
>>
>> --Guido (if I don't respond to this thread for the next two weeks,
>> it's because I'm on vacation :-)
>>
>>
>> On Sat, Dec 21, 2013 at 5:37 PM, Ben Darnell <ben at bendarnell.com> wrote:
>> > On Sat, Dec 21, 2013 at 7:26 PM, Guido van Rossum <guido at python.org>
>> wrote:
>> >>
>> >> On Sat, Dec 21, 2013 at 2:48 PM, Sergii Mikhtoniuk <
>> mikhtonyuk at gmail.com>
>> >> wrote:
>> >> > Indeed there is a lot of overlap between asyncio and
>> concurrent.futures
>> >> > packages, so it would be very interesting to hear your overall
>> thoughts
>> >> > on
>> >> > role/future of concurrent package. Do you consider it rudimentary and
>> >> > replaceable by asyncio.Future completely?
>> >>
>> >> They don't really compare. concurrent.futures is about *threads*.
>> >> asyncio.Future is about *avoiding* threads in favor of more
>> >> lightweight "tasks" and "coroutines", which in turn are built on top
>> >> of lower-level callbacks.
>> >
>> >
>> > concurrent.futures.ThreadPoolExecutor is about threads; the Future class
>> > itself is broader.  When I integrated Futures into Tornado I used
>> > concurrent.futures.Future directly (when available).  asyncio.Future is
>> just
>> > an optimized version of c.f.Future (the optimization comes from assuming
>> > single-threaded usage).  There should at least be a common ABC between
>> them.
>> >
>> >>
>> >>
>> >> (While I want to get away from callbacks as a programming paradigm,
>> >> asyncio uses them at the lower levels both because they are a logical
>> >> low-level building block and for interoperability with other
>> >> frameworks like Tornado and Twisted.)
>> >>
>> >> > I think the question is even not much about which is your preferred
>> >> > implementation, but rather do we want having futures as stand-alone
>> >> > package
>> >> > or not. Do you see all these implementations converging in future?
>> >>
>> >> I see them as not even competing. They use different paradigms and
>> >> apply to different use cases.
>> >>
>> >> > To me Future is a very simple and self-contained primitive,
>> independent
>> >> > of
>> >> > thread pools, processes, IO, and networking.
>> >>
>> >> (Agreed on the I/O and networking part only.)
>> >>
>> >> > One thing concurrent.futures
>> >> > package does a very good job at is defining an isolated namespace for
>> >> > futures, stressing out this clear boundary (let’s disregard here
>> >> > ThreadPoolExecutor and ProcessPoolExecutor classes which for some
>> reason
>> >> > ended up in it too).
>> >>
>> >> Actually the executor API is an important and integral part of that
>> >> package, and threads underlie everything you can do with its Futures.
>> >>
>> >> > So when I think of schedulers and event loops implementations I see
>> them
>> >> > as
>> >> > ones that build on top of the Future primitive, not providing it as
>> part
>> >> > of
>> >> > their implementation. What I think is important is that having
>> unified
>> >> > Future class simplifies interoperability between different kinds of
>> >> > schedulers, not necessarily associated with asyncio event loops
>> (process
>> >> > pools for example).
>> >>
>> >> The interoperability is completely missing. I can't tell if you've
>> >> used asyncio at all, but the important operation of *waiting* for a
>> >> result is fundamentally different there than in concurrent.futures. In
>> >> the latter, you just write "x = f.result()" and your thread blocks
>> >> until the result is available. In asyncio, you have to write "x =
>> >> yield from f.result()" which is a coroutine block that lets other
>> >> tasks run in the same thread. ("yield from" in this case is how Python
>> >> spells the operation that C# calls "await").
>> >
>> >
>> > The way I see it, the fundamental operation on Futures is
>> add_done_callback.
>> > We then have various higher-level operations that let us get away from
>> using
>> > callbacks directly.  One of these happens to be a method on Future: the
>> > blocking mode of Future.result().  Another is implemented in asyncio and
>> > Tornado, in the ability to "yield" a Future.  The blocking mode of
>> result()
>> > is just a convenience; if asyncio-style futures had existed first then
>> we
>> > could instead have a function like "x =
>> concurrent.futures.wait_for(f)".  In
>> > fact, you could write this wait_for function today in a way that works
>> for
>> > both concurrent and asyncio futures.
>> >
>> > This is already interoperable:  Tornado's Resolver interface
>> > (
>> https://github.com/facebook/tornado/blob/master/tornado/netutil.py#L184)
>> > returns a Future, which may be generated by a ThreadPoolExecutor or an
>> > asynchronous wrapper around pycares or twisted.  In the other direction
>> I've
>> > worked on hybrid apps that have one Tornado thread alongside a bunch of
>> > Django threads; in these apps it would work to have a Django thread
>> block on
>> > f.result() for a Future returned by Tornado's AsyncHTTPClient.
>> >
>> > -Ben
>> >
>> >>
>> >> > Futures are definitely not the final solution for concurrency problem
>> >> > but
>> >> > rather a well-established utility for representing async operations.
>> It
>> >> > is a
>> >> > job of higher layer systems to provide more convenient ways for
>> dealing
>> >> > with
>> >> > asyncs (yield from coroutines, async/await rewrites etc.),
>> >>
>> >> But unless you are proposing some kind of radical change to add
>> >> compile-time type checking/inference to Python, the rewrite option is
>> >> unavailable in Python.
>> >>
>> >> > so I would not
>> >> > say that futures encourage callback-style programming,
>> >>
>> >> The concurrent.futures.Future class does not. But unless I misread
>> >> your proposal, your extensions do.
>> >>
>> >> > it’s simply a
>> >> > lower-layer functionality. On the contrary, monadic
>> >>
>> >> (Say that word one more time and everyone tunes out. :-)
>> >>
>> >> > methods for futures
>> >> > composition (e.g. map(), all(), first() etc.) ensure that no errors
>> >> > would be
>> >> > lost in the process, so I think they would complement yield from
>> model
>> >> > quite
>> >> > nicely by hiding complexity of state maintenance from user and
>> reducing
>> >> > the
>> >> > number of back-and-forth communications between event loop and
>> >> > coroutines.
>> >>
>> >> I'm not sure I follow. Again, I'm not sure if you've actually written
>> >> any code using asyncio.
>> >>
>> >> TBH I've written a fair number of example programs for asyncio and
>> >> I've very rarely felt the need for these composition functions. The
>> >> main composition primitive I tend to use is "yield from".
>> >>
>> >> > Besides Futures, reactive programming
>> >>
>> >> What *is* reactive programming? If you're talking about
>> >> http://en.wikipedia.org/wiki/Reactive_programming,
>> >> I'm not sure that it maps well to Python.
>> >>
>> >> > has more utilities to offer, such as
>> >> > Observables (representing asynchronous streams of values). It is
>> also a
>> >> > very
>> >> > useful abstraction with a rich set of composition strategies
>> (merging,
>> >> > concatenation, grouping), and may deserve its place in separate
>> package.
>> >>
>> >> It all sounds very abstract and academic. :-)
>> >>
>> >> > Hope to hear back from you to get better picture on overall design
>> >> > direction
>> >> > here before jumping to implementation details.
>> >> >
>> >> > Brief follow-up to your questions:
>> >> >  - Idea behind the Future/Promise separation is to draw a clean line
>> >> > between
>> >> > client-facing and scheduler-side APIs respectively. Futures should
>> not
>> >> > expose any completion methods, which clients should not call anyway.
>> >>
>> >> Given that Future and Promise are often used synonymously, using them
>> >> to make this distinction sounds confusing. I agree that Futures have
>> >> two different APIs, one for the consumer and another for the producer.
>> >> But I'm not sure that it's necessary to separate them more strictly --
>> >> convention seems good enough to me here. (It's the same with many
>> >> communication primitives, like queues and even threads.)
>> >>
>> >> (The one thing that trips people up frequently is that, while
>> >> set_result() and set_exception() are producer APIs, cancel() is a
>> >> consumer API, and the cancellation signal travels from the consumer to
>> >> the producer.)
>> >>
>> >> >  - Completely agree with you that Twisted-style callbacks are evil
>> and
>> >> > it is
>> >> > better to have single code path for getting result or raising
>> exception
>> >> >  - Sorry for bad grammar in the proposal, it’s an early draft
>> written in
>> >> > 3
>> >> > AM, so I will definitely improve on it if we decide to move forward.
>> >>
>> >> No problem!
>> >>
>> >> > Thanks,
>> >> > Sergii
>> >>
>> >> --
>> >> --Guido van Rossum (python.org/~guido)
>> >> _______________________________________________
>> >> Python-ideas mailing list
>> >> Python-ideas at python.org
>> >> https://mail.python.org/mailman/listinfo/python-ideas
>> >> Code of Conduct: http://python.org/psf/codeofconduct/
>> >
>> >
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20131222/6671ebad/attachment-0001.html>