I have started work on a PEP for the inclusion of the PyProcessing module (http://pypi.python.org/pypi/processing/ and http://developer.berlios.de/projects/pyprocessing) for inclusion into the stdlib in an upcoming release.

The pyprocessing module "mostly mimicks" the threading module API to provide a "drop in" process-based approach to concurrency allowing applications within python to utilize multiple cores. For example:

from threading import Thread class threads_object(Thread): def run(self): function_to_run()

becomes:

from processing import Process class process_object(Process): def run(self): function_to_run()

Currently, the module runs on Unix/Linux/OSX and Windows. It supports the following features:

* Objects can be transferred between processes using pipes or multi-producer/multi-consumer queues.

* Objects can be shared between processes using a server process or (for simple data) shared memory.

* Equivalents of all the synchronization primitives in ``threading`` are available.

* A ``Pool`` class makes it easy to submit tasks to a pool of worker processes.

In addition to local concurrency ala "just like threading" model - the processing module allows users to also share data and processes across a cluster of machines via the server Managers and Proxy objects (secured data transfer is supported, data is transferred via pickles). I believe that prior/during inclusion, additional tests will be desired to enhance coverage - but the tests which are already provided test and showcase the module well.

I have spoken to the writer, Richard Oudkerk about his willingness to maintain the processing module according to normal stdlib requirements and he is more than willing to do so.

I believe inclusion into the standard library will be very beneficial for a lot of people looking to larger-scale applications and for a method to side-step the current threading/GIL implementation. This module easily allows users to exploit their "$N core" machines, in a fashion they are already familiar with.

I would suggest that the module is placed at the top-level next to the threading module, however, there is the thought that both this module, and the threading module should be moved to a concurrent.* namespace (i.e: concurrent.threading, concurrent.processing) to allow for additional library inclusion at a later date.

IMHO: I think this is simply a "first step" in the evolution of the python stdlib to support these sorts of things - but I also believe it is an excellent first step.

Please feel free to ask any questions, comments, etc. The more feedback the better the PEP will be!

-jesse