[Python-ideas] .then execution of actions following a future's completion

[Daniel Collins]

So, just going point by point:

Yes, absolutely put this off for 3.8. I didn’t know the freeze was so close or I would have put the 3.8 tag on originally.

Yes, absolutely it is only meant for concurrent.futures futures, it only changes async where async uses concurrent.futures futures.

Here’s a more fleshed out description of the use case:

Assume you have two functions. Function a(x: str)->AResult fetches an AResult object from a web resource, function b(y: AResult) performs some computationally heavy work on AResult.

Assume you’re calling a 10 times with a threadpoolexecutor with 2 worker theads.  If you were to schedule a as future using submit, and b as a callback, the executions would look like this:

ExecutorThread: b*10
Worker1: a*5
Worker2: a*5

This only gets worse as more work (b) is scheduled as a callback for the result from a.

Now you could resolve this by, instead of submitting b as a callback, submitting the following lambda:

lambda x: executor.submit(b, x) 

But then you wouldn’t have easy access to this new future. You would have to build a lot of boilerplate code to collect that future into some external collection, and this would only get worse the deeper the nesting goes.

With this syntax on the other hand, if you run a 10 times using submit, but then run a_fut.then(b) for each future, execution instead looks like this:

ExecutorThread: 
Worker1: a*5 b*5
Worker2: a*5 b*5

You can also do additional depth easily. Suppose you want to run 3 c operations (processes the output of b) for each b operation. Then you could call this like

b_fut = a_fut.then(b)

for i in range(3):
    b_fut.then(c)

And the execution would look like this:

ExecutorThread:
Worker1: a*5 b*5 c*15
Worker2: a*5 b*5 c*15

Which would be very difficult to do otherwise, and distributes the load across the workers, while having direct access to the outputs of the calls to c.

-dancollins34

Sent from my iPhone

> On Jan 26, 2018, at 1:07 AM, Guido van Rossum <guido at python.org> wrote:
> 
> I really don't want to distract Yury with this. Let's consider this (or something that addresses the same need) for 3.8.
> 
> To be clear this is meant as a feature for concurrent.futures.Future, not for asyncio.Future. (It's a bit confusing since you also change asyncio.)
> 
> Also to be honest I don't understand the use case *or* the semantics very well. You have some explaining to do...
> 
> (Also, full links: https://bugs.python.org/issue32672; https://github.com/python/cpython/pull/5335)
> 
>> On Thu, Jan 25, 2018 at 8:38 PM, Daniel Collins <dancollins34 at gmail.com> wrote:
>> Hello all,
>> 
>> So, first time posting here. I’ve been bothered for a while about the lack of the ability to chain futures in python, such that the next future will execute upon the first’s completion.  So I submitted a pr to do this.  This would add the .then(self, fn) method to concurrent.futures.Future.  Thoughts?
>> 
>> -dancollins34
>> 
>> Github PR #5335
>> bugs.python.org issue #32672
>> 
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>> 
> 
> 
> 
> -- 
> --Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180126/6e42e6d6/attachment.html>