[Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Dj Gilcrease]

On Mon, Mar 8, 2010 at 12:04 PM, Dj Gilcrease <digitalxero at gmail.com> wrote:
> A style I have used in my own code in the past is a Singleton class
> with register and create methods, where the register takes a
> name(string) and the class and the create method takes the name and
> *args, **kwargs and acts as a factory.


So I decided to play with this design a little and since I made it a
singleton I decided to place all the thread/process tracking and exit
handle code in it instead of having the odd semi-global scoped
_shutdown, _thread_references, _remove_dead_thread_references and
_python_exit objects floating around in each executor file, seems to
work well. The API would be

from concurrent.futures import executors

executor = executors.create(NAME, *args, **kwargs) # NAME is 'process'
or 'thread' by default


To create your own executor you create your executor class and add the
following at the end

from concurrent.futures import executors, ExecutorBase
class MyExecutor(ExecutorBase): ...
executors.register(NAME, MyExecutor)

It checks to see if your executor is a subclass of ExecutorBase, but
only does a UserWarning if it is not since you should know what
methods are required to be an executor like object, so if you are not
subclassing ExecutorBase you should suppress the UserWarning before
you register you class and un-suppress it after


Some Helper Methods/Properties on the executors Singleton
add_joinable_ref - This replaces the _thread_references.add system of
tracking threads, and it allows for adding processes as well hence the
joinable_ref name. It does check to make sure the ref being passes has
a join method then creates a weakref to it and adds it to a set. For
every thread or process your executor creates you should call this
with so it will be tracked properly

cleanup_joinable_refs - This replaces the
_remove_dead_thread_references that had to be written for each
executor individually. This should be called periodically, currently
it is only called when you create a new executor since it is a
blocking method (it uses a thread lock to make sure the set of
references does not change while it is discarding old ones)

shutdown - is a readonly property and replaces the _shutdown global
var that had to be created for each executor individually, it is set
in the executors destructor

__del__ - replaces the _python_exit method that had to be written for
each executor individually



If this API change isnt accepted its no big deal, since I am only
changing it due to personal pet peeves and the only real issue I had
with the package was scoping which has already been addressed by
adding it to a concurrent package