[stdlib-sig] futures - a new package for asynchronous execution

[Brian Quinlan]

Hey Antoine,

Sorry for not getting back to you sooner - I actually thought that I  
did reply but I see now that I didn't.

On 14 Nov 2009, at 01:22, Antoine Pitrou wrote:

>
> Hey,
>
>> Future remains unchanged - I disagree that Deferreds would be better,
>> that .exception() is not useful, and that .result() should be
>> renamed .get() or .__call__().
>
> On what grounds do you disagree with the latter?

It feels hacky. Getting the result doesn't feel so special that it  
deserves to be a call rather than a simple getter.

> Another question: is the caught exception an attribute of the future?

Yes

> If
> so, is there any mechanism to clean it up (and its traceback) once the
> future has been "consumed"?

No there isn't. That's a good point though. I wonder if futures will  
tend to long-lived after there results are available?

>
>> map becomes a utility function:
>>
>> def map(executor, *iterables, timeout=None)
>
> Why? map() can be defined on the ABC, so that subclasses don't have to
> provide their own implementation.
>
> An utility function which looks like a method and shadows the name  
> of a
> built-in looks like a bad choice to me.

Good point.

>> wait becomes a utility function that can wait on any iterable of
>> Futures:
>>
>> def wait(futures, return_when=ALL_COMPLETED)
>
> Does it work if the futures are executed by different executors?
> If not, it should be an Executor method.

Yes, it goes.

>
>> return_when indicates when the method should return. It must be one  
>> of
>> the following constants:
>>
>>     NEXT_COMPLETED
>>     NEXT_EXCEPTION
>>     ALL_COMPLETED
>
> Can you outline the difference between NEXT_COMPLETED and
> NEXT_EXCEPTION? What happens if I ask for NEXT_COMPLETED but the next
> future to complete raises an exception? etc.

NEXT_COMPLETED includes futures that raise. Completed in this sense  
means "done running".

>
>> def itercompleted(futures, timeout=None):
>>
>> Returns an iterator that returns a completed Future from the given
>> list when __next__() is called. If no Futures are completed then
>> __next__() is called then __next__() waits until one does complete.
>
> What about futures which complete with an exception?

They are included.


>> with futures.ThreadPoolExecutor(50) as executor:
>>   fs = [executor.submit(load_url, url, timeout=30) for url in URLS]
>
> The use of "with" here still is counter-intuitive, because it does not
> clean up resources immediately as it would seem to do. "with" is  
> always
> synchronous in other situations.

Maybe waiting until all pending futures are done executing would be  
better.

>
>> What do you think? Are we moving in the right direction?
>
> Perhaps, yes, but there are still lots of dark areas.
>
> Besides, it's obvious that the package has to mature, and should be
> tested by other people.


It would be great if other people tested the API. I'm not sure what  
you mean by "mature" though.

Cheers,
Brian