[Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Brian Quinlan]

On 6 Mar 2010, at 17:50, Phillip J. Eby wrote:

> At 01:19 AM 3/6/2010, Jeffrey Yasskin wrote:
>> On Fri, Mar 5, 2010 at 10:11 PM, Phillip J. Eby <pje at telecommunity.com 
>> > wrote:
>> > I'm somewhat concerned that, as described, the proposed API ...  
>> [creates] yet another alternative (and
>> > mutually incompatible) event loop system in the stdlib ...
>>
>> Futures are a blocking construct; they don't involve an event loop.
>
> And where they block is in a loop, waiting for events (completed  
> promises) coming back from other threads or processes.
>
> The Motivation section of the PEP also stresses avoiding reinvention  
> of such loops, and points to the complication of using more than one  
> at a time as a justification for the mechanism.  It seems relevant  
> to at least address why wrapping multiprocessing and multithreading  
> is appropriate, but *not* dealing with any other form of sync/async  
> boundary, *or* composition of futures.
>
> On which subject, I might add, the PEP is silent on whether  
> executors are reentrant to the called code.  That is, can I call a  
> piece of code that uses futures, using the futures API?  How will  
> the called code know what executor to use?  Must I pass it one  
> explicitly?  Will that work across threads and processes, without  
> explicit support from the API?

Executors are reentrant but deadlock is possible. There are two  
deadlock examples in the PEP.

>
> IOW, as far as I can tell from the PEP, it doesn't look like you can  
> compose futures without *global* knowledge of the application...   
> and in and of itself, this seems to negate the PEP's own motivation  
> to prevent duplication of parallel execution handling!
>
> That is, if I use code from module A and module B that both want to  
> invoke tasks asynchronously, and I want to invoke A and B  
> asynchronously, what happens?  Based on the design of the API, it  
> appears there is nothing you can do except refactor A and B to take  
> an executor in a parameter, instead of creating their own.

A and B could both use their own executor instances. You would need to  
refactor A and B if you wanted to manage thread and process counts  
globally.

> It seems therefore to me that either the proposal does not define  
> its scope/motivation very well, or it is not well-equipped to  
> address the problem it's setting out to solve.  If it's meant to be  
> something less ambitious -- more like a recipe or example -- it  
> should properly motivate that scope.  If it's intended to be a  
> robust tool for composing different pieces of code, OTOH, it should  
> absolutely address the issue of writing composable code...  since,  
> that seems to be what it says the purpose of the API is.  (I.e.,  
> composing code to use a common waiting loop.)

My original motivation when designing this module was having to deal  
with a lot of code that looks like this:

def get_some_user_info(user):
   x = make_ldap_call1(user)
   y = make_ldap_call2(user)
   z = [make_db_call(user, i) for i in something]

   # Do some processing with x, y, z and return a result

Doing these operations serially is too slow. So how do I parallelize  
them? Using the threading module is the obvious choice but having to  
create my own work/result queue every time I encounter this pattern is  
annoying. The futures module lets you write this as:

def get_some_user_info(user):
   with ThreadPoolExecutor(max_threads=10) as executor:
     x_future = executor.submit(make_ldap_call1, user)
     y_future = executor.submit(make_ldap_call2, user)
     z_futures = [executor.submit(make_db_call, user, i) for i in  
something]
    finished, _ = wait([x_future, y_future] + z_futures,  
return_when=FIRST_EXCEPTION)
  for f in finished:
    if f.exception():
      raise f.exception()
  x = x_future.result()
  y = y_future.result()
  z = [f.result() for f in z_futures]

   # Do some processing with x, y, z and return a result

> And, existing Python async APIs (such as Twisted's Deferreds)  
> actually *address* this issue of composition; the PEP does not.   
> Hence my comments about not looking at existing implementations for  
> API and implementation guidance.  (With respect to what the API  
> needs, and how it needs to do it, not necessarily directly copying  
> actual APIs or implementations.  Certainly some of the Deferred API  
> naming has a rather, um, "twisted" vocabulary.)

Using twisted (or any other asynchronous I/O framework) forces you to  
rewrite your I/O code. Futures do not.

Cheers,
Brian