[Python-ideas] PEP draft - Composable futures for reactive programming

[Sergii Mikhtoniuk]

Hello Guido, Nick,

Thanks a lot for your valuable feedback.

Indeed there is a lot of overlap between *asyncio* and
*concurrent.futures*packages, so it would be very interesting to hear
your overall thoughts on
role/future of *concurrent* package. Do you consider it rudimentary and
replaceable by *asyncio.Future* completely?

I think the question is even not much about which is your preferred
implementation, but rather do we want having futures as stand-alone package
or not. Do you see all these implementations converging in future?

To me *Future* is a very simple and self-contained primitive, independent
of thread pools, processes, IO, and networking. One thing
*concurrent.futures* package does a very good job at is defining an
isolated namespace for futures, stressing out this clear boundary (let’s
disregard here *ThreadPoolExecutor* and *ProcessPoolExecutor* classes which
for some reason ended up in it too).

So when I think of *schedulers* and *event loops* implementations I see
them as ones that build on top of the *Future* primitive, not providing it
as part of their implementation. What I think is important is that having
unified *Future* class simplifies interoperability between different kinds
of schedulers, not necessarily associated with *asyncio* event loops
(process pools for example).

*Futures* are definitely not the final solution for concurrency problem but
rather a well-established utility for representing async operations. It is
a job of higher layer systems to provide more convenient ways for dealing
with asyncs (*yield from* coroutines, *async/await* rewrites etc.), so I
would not say that futures encourage callback-style programming, it’s
simply a lower-layer functionality. On the contrary, monadic methods for
futures composition (e.g. *map()*, *all()*, *first()* etc.) ensure that no
errors would be lost in the process, so I think they would complement*
yield from* model quite nicely by hiding complexity of state maintenance
from user and reducing the number of back-and-forth communications between
event loop and coroutines.

Besides *Futures*, reactive programming has more utilities to offer, such
as *Observables* (representing asynchronous streams of values). It is also
a very useful abstraction with a rich set of composition strategies
(merging, concatenation, grouping), and may deserve its place in separate
package.

Hope to hear back from you to get better picture on overall design
direction here before jumping to implementation details.

Brief follow-up to your questions:
 - Idea behind the Future/Promise separation is to draw a clean line
between client-facing and scheduler-side APIs respectively. Futures should
not expose any completion methods, which clients should not call anyway.
 - Completely agree with you that Twisted-style callbacks are evil and it
is better to have single code path for getting result or raising exception
 - Sorry for bad grammar in the proposal, it’s an early draft written in 3
AM, so I will definitely improve on it if we decide to move forward.


Thanks,
Sergii


On Sat, Dec 21, 2013 at 7:59 PM, Guido van Rossum <guido at python.org> wrote:

> I'm pretty worried about this proposal. It seems to encourage
> callback-style programming, whereas I would like to get away from it
> (that's the main thrust of asyncio/PEP 3156). The proposal reminds me
> of Twisted Deferred, which is too complicated for my taste.
>
> Note that asyncio has a composition operation of its own, gather(),
> but encourages using standard try/except/else/finally syntax for error
> handling, rather than adding Twisted-style errbacks.
>
> Nevertheless it is possible that I am misunderstanding the proposal
> (e.g. the distinction between Future and Promise wasn't clear from
> reading the draft PEP, nor how it would interact with asyncio). I
> recommend that the author work find a way to improve the English
> grammar of the proposal, e.g. by working with a native speaker.
>
> On Sat, Dec 21, 2013 at 6:12 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > On 21 December 2013 23:14, Sergii Mikhtoniuk <mikhtonyuk at gmail.com>
> wrote:
> >> Please have a look at this PEP draft, and reference implementation (as
> >> separate library).
> >>
> >> I’m very interested in:
> >> - How PEPable is this?
> >
> > This looks like a really interesting idea, and well worth pursuing as
> > a PEP for 3.5 (the failure to make the Future/Promise split was
> > actually noted as a flaw in the concurrent.futures design at the time,
> > but wasn't seen as serious enough to prevent inclusion of the module.
> > However, this draft PEP does a decent job of showing the kinds of
> > operations that the current combined class design can make difficult)
> >
> >> - What are your thoughts on backward compatibility (current
> implementation
> >> does not sacrifice any design points for it, but better compatibility
> can be
> >> achieved)?
> >
> > Yes, I think better compatibility is needed. One possible option would
> > be to take the path of defining a "concurrent.futures.abc" module that
> > provides this higher level API, and largely leave the existing
> > concrete classes alone (aside from adjusting them to fit the ABC).
> >
> > The main possible issue I see is with the result setting APIs on the
> > existing Future objects.
> >
> > There are also a lot of staticmethod declarations that look like they
> > should probably be classmethod declarations.
> >
> > Cheers,
> > Nick.
> >
> > --
> > Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> > _______________________________________________
> > Python-ideas mailing list
> > Python-ideas at python.org
> > https://mail.python.org/mailman/listinfo/python-ideas
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20131222/5f8415d4/attachment-0001.html>