[Python-Dev] futures API

[Raymond Hettinger]

On Dec 9, 2010, at 9:02 AM, Brian Quinlan wrote:

> 
> On Dec 9, 2010, at 4:26 AM, Thomas Nagy wrote:
> 
>> Hello,
>> 
>> I am looking forward to replacing a piece of code (http://code.google.com/p/waf/source/browse/trunk/waflib/Runner.py#86) by the futures module which was announced in python 3.2 beta. I am a bit stuck with it, so I have a few questions about the futures:
>> 
>> 1. Is the futures API frozen?
> 
> Yes.

Yes, unless the current API is defective in some way.  A beta1 release is a chance for everyone to exercise the new API and discover whether it is problematic in any real world applications.

> 
>> 2. How hard would it be to return the tasks processed in an output queue to process/consume the results while they are returned? The code does not seem to be very open for monkey patching.
> 
> You can associate a callback with a submitted future. That callback could add the future to your queue.

The would be a good example for the docs.

> 
>> 3. How hard would it be to add new tasks dynamically (after a task is executed) and have the futures object never complete?
> 
> I'm not sure that I understand your question. You can submit new work to an Executor at until time until it is shutdown and a work item can take as long to complete as you want. If you are contemplating tasks that don't complete then maybe you could be better just scheduling a thread.
> 
>> 4. Is there a performance evaluation of the futures code (execution overhead) ?
> 
> No. Scott Dial did make some performance improvements so he might have a handle on its overhead.

FWIW, the source code is short and readable.  From my quick read, it looks to be a relatively thin wrapper/adapter around existing tools.  Most of the work still gets done by the threads or processes themselves.  Think of this as a cleaner, more centralized API around the current toolset -- there is no deep new technology under the hood.

Raymond