[stdlib-sig] futures - a new package for asynchronous execution

[Jesse Noller]

On Thu, Feb 25, 2010 at 10:28 PM, Brian Quinlan <brian at sweetapp.com> wrote:
>
> On Feb 26, 2010, at 5:49 AM, Jesse Noller wrote:
...
>> Yes; I think this needs to be part of a new "concurrent" package in
>> the stdlib - e.g. concurrent.futures, understanding things within
>> multiprocessing will be put in there shortly, and possibly other
>> things such as a threadpool and other common sugary abstractions.
>
>
> Are you imagining that futures would be subpackage of concurrent with a
> single logical namespace i.e.
>
> concurrent/
>  __init__.py
>  futures/
>    __init__.py
>   threads.py
>   processes.py
>   ...
>
> from concurrent.future import wait
> from concurrent.future import ThreadPoolExecutor
>
> Or should the futures package be merged into the concurrent package i.e.
>
> concurrent/
>  __init__.py
>  futures.py
>  threadpoolexecutor.py (was threads.py)
>  processpoolexecutor.py (as processes.py)
>
> from concurrent.future import wait
> from concurrent.future.threadpoolexecutor import ThreadPoolExecutor
>

I'm on the fence. I took a few minutes to think about this today, and
my gut says concurrent with a single logical namespace - so:

from concurrent import futures
futures.ThreadPoolExecutor

And so on. Others might balk at a deeper namespace, but then say we add:

concurrent/
    futures/
    pool.py (allows for a process pool, or threadpool)
    managers.py

And so on. I'm trying to mentally organize things to "be like"
java.util.concurrent [1] - ideally we could move/consolidate the
common sugar into this package, and remove the other "stuff" from
multiprocessing as well. That way multiprocessing can become "just"
Process and the locking stuff, ala threading, and the rest of the
other nice-things can be made to work with threads *and* processes ala
what you've done with futures.

This is just a thought; I've been thinking about it a lot, but I admit
not having sat down and itemized the things that would live in this
new home. The futures discussion just spurred me to propose the idea
sooner rather than later.

Jesse

[1] http://java.sun.com/javase/6/docs/api/java/util/concurrent/package-summary.html