Mailman 3 November 2009 - stdlib-sig

Re: [stdlib-sig] futures - a new package for asynchronous execution
by Laura Creighton Nov. 7, 2009

Nov. 7, 2009

Anybody here looked at Kamaelia? http://www.kamaelia.org/ I haven't, but wonder if we are considering re-inventing the wheel. Laura

4 4

comments on PEP for futures - execute computations asynchronously
by Jim Jewett Nov. 7, 2009

Nov. 7, 2009

(1) It would be helpful to have the API classes named earlier, even before the examples if the examples take more than three lines (as they currently do). The details of what methods they have can come later, but naming the classes should serve as a useful landmark. (2) Within the FutureList methods definitions, please refer to the FutureList, rather than "the list". The first few times I came across that, I was trying to figure out how you knew the container would be a list, or whether this was a list that the caller had created. I then kept trying to view methods like has_exception_futures as global functions that ought to take a list of futures as an argument. (3) Why even have a FutureList? Can't the Executor object itself act as a container for the tasks it is executing? (And you might want a way to remove tasks from the Executor, short of shutting down the whole thing.) (4) Please add some sort of factory or global function so that users can say "parallelize this however makes sense" and let the environment pick. (For example, Jython would presumably push threads, as would windows, but Linux implementations might be happier with processes.) -jJ

1 0

Re: [stdlib-sig] Concurrent/Jython (was: futures - a new package for asynchronous execution)
by Jesse Noller Nov. 7, 2009

Nov. 7, 2009

On Sat, Nov 7, 2009 at 11:37 AM, Frank Wierzbicki <fwierzbicki(a)gmail.com> wrote: > If it can be done in pure Python I'd certainly be up for taking a a > crack at such a patch. If it involves significant work with C and > threading it might be a little out of my scope. If pure python is > out, I may end up implementing those parts missing in threading.py in > Java for Jython, and then circling back to see if doing it in C for > CPython makes sense. > > -Frank Figured I'd start a new thread rather than overloading the existing one. If we were to seriously go down the path of building out a concurrent package for Python's stdlib, I think 99% of the work would be in pure python. Right now multiprocessing and threading expose the primitives needed to do most of the work without diving into C. For example, the pool.py module [1] multiprocessing has could easily have threads swapped in - this would probably change the API to look like this: Original: class Pool(object): ''' Class which supports an async version of the `apply()` builtin ''' Process = Process def __init__(self, processes=None, initializer=None, initargs=()): New: class Pool(object): ''' Class which supports an async version of the `apply()` builtin ''' worker = threading.Thread def __init__(self, worker_type=multiprocessing.Process, workers=None, initializer=None, initargs=()): Theres some other internals which would need to be adjusted, but just modifying it to support passing in the Worker type to use would make it easy to swap it for threads vs. process. My person preference would be to have the above hidden with _ and two classes: class ThreadPool class ProcessPool Which just called the hidden _Pool with the worker type, but that's a smell-thing. I think explicitness in APIs dealing with threads and processes is preferable than highly flexible generic APIs (the restriction on multiprocessing being "objects must be pickleable"). Then again, passing in the worker type means less work for IronPython/Jython - users could just be told "just accept the default thread worker type" - they would only need to change process-based code to remove the worker_type argument. Or Jython could swap it under the covers. So, my pony list would look something like: 1> add concurrent 2> Put futures in concurrent 3> Refactor multiprocessing.pool into concurrent, add deprecation notes to the multiprocessing APIs in the docs A simple producer/consumer implementation could also be added in here too (one day). This work should only target Python 3, imho. jesse [1]: http://svn.python.org/view/python/trunk/Lib/multiprocessing/pool.py?revisio…

1 0

Re: [stdlib-sig] futures - a new package for asynchronous execution
by Brian Quinlan Nov. 7, 2009

Nov. 7, 2009

On 7 Nov 2009, at 18:37, inhahe wrote: > i don't understand this at all > i hope you will provide a basic explanation of what we're doing for > us simpletons :P > Sure, but next time could you ask a more precise question? ;-) # Create a pool of threads to execute calls. with futures.ThreadPoolExecutor(max_threads=5) as executor: # Schedule the given calls to run using the thread pool created above and # return a FutureList (a list of Futures + some convenience methods). Called # without the "return_when" argument, waits until all calls are complete. futures_list = executor.run_to_futures(...) # Iterate through every Future. A Future represents one of the asynchronous # calls. for url, future in zip(URLS, future_list): # Check if the call raised an exception. if future.exception() is not None: # Print the exception print('%r generated an exception: %s' % (url, future.exception())) else: # The call returned successfully so future.result() contains the return # value. print('%r page is %d bytes' % (url, len(future.result()))) Cheers, Brian > import futures > import functools > import urllib.request > > URLS = [ > 'http://www.foxnews.com/', > 'http://www.cnn.com/', > 'http://europe.wsj.com/', > 'http://www.bbc.co.uk/', > 'http://some-made-up-domain.com/'] > > def load_url(url, timeout): > return urllib.request.urlopen(url, timeout=timeout).read() > > # Use a thread pool with 5 threads to download the URLs. Using a pool > # of processes would involve changing the initialization to: > # with futures.ProcessPoolExecutor(max_processes=5) as executor > with futures.ThreadPoolExecutor(max_threads=5) as executor: > future_list = executor.run_to_futures( > [functools.partial(load_url, url, 30) for url in URLS]) > > # Check the results of each future. > for url, future in zip(URLS, future_list): > if future.exception() is not None: > print('%r generated an exception: %s' % (url, > future.exception())) > else: > print('%r page is %d bytes' % (url, len(future.result()))) > > In this example, executor.run_to_futures() returns only when every > url has been retrieved but it is possible to return immediately, on > the first completion or on the first failure depending on the > desired work pattern. > > The complete docs are here: > http://sweetapp.com/futures/ >

1 0