[stdlib-sig] futures - PEP and API

[Brian Quinlan]

On 8 Nov 2009, at 03:13, Antoine Pitrou wrote:

>
>> The PEP is right here:
>>
>> http://code.google.com/p/pythonfutures/source/browse/trunk/PEP.txt
>>
>> I'm interested in hearing specific complaints about the API in the
>> context of what it's trying to *do*. The only thing which jumped out
>> at me was the number of methods on FutureList;
>
> Yes that would be the first complaint. Then many of those methods  
> are as
> (un)trustable as, say, Queue.qsize().
>
> An example :
>        """`done_futures()`
>
>        Return an iterator over all `Future` instances that completed  
> or
>        were cancelled."""
>
> First, it claims to return an iterator but the internal container  
> could
> mutate while iterating (since it can be mutated when a task terminates
> in another thread). So the API looks broken with respect to what the
> semantics dictate. It should probably return a distinct container  
> (list
> or set) instead.
> Second, by the time the result is processed by the caller, there's no
> way to know if the information is still valid or not. It's entirely
> speculative, which makes it potentially deceiving -- and should be
> mentioned in the doc.
>
>        """`has_done_futures()`
>
>        Return `True` if any `Future` in the list has completed or was
>        successfully cancelled."""
>
> Same problem. Please note that it can be removed if `done_futures()`
> returns a container, since you then just have to do a boolean check on
> the container (that would remove 5 methods :-)).
>
>
> Then about the Future API itself. I would argue that if we want it  
> to be
> a simple helper, it should be as simple to use as a weakref.
>
> That is, rather than :
>
>        """`result(timeout=None)`
>
>        Return the value returned by the call.
>        [...]
>
>        `exception(timeout=None)`
>
>        Return the exception raised by the call."""
>
> Make the object callable, such as `future(timeout=None)` either  
> returns
> the computed result (if successful), raises an exception (if failed)  
> or
> raises a TimeoutError.
>
>
> Then about the Executor API. I don't understand why we have the
> possibility to wait on a FutureList *and* on the Executor's
> run_to_results() method. I think all wait-type methods should be  
> folded
> in to the Future or FutureList API, and the Executor should only
> generate that Future(List).
>
> Practically, there should be two ways to wait for multiple results,
> depending on whether you need the results ordered or not. In the web
> crawling situation given as example, it is silly to wait for the  
> results
> in order rather than process each result as soon as it gets available.
> (*)
>
> I don't understand why the Executor seems to be used as a context
> manager in the examples. Its resources are still alive after the  
> "with"
> since the tasks are still executin, so it can't possibly have  
> cleaned up
> anything, has it?

The executor needs to know when no more futures will be scheduled so  
it can shutdown its thread/process pool after it has finished the  
current work items.

Let me think about your other comments for a while.

Cheers,
Brian



>
> (*) And, of course, you start to understand why a callback-based API
> such as Deferreds makes a lot of sense...
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> stdlib-sig mailing list
> stdlib-sig at python.org
> http://mail.python.org/mailman/listinfo/stdlib-sig