[Python-ideas] Parallel processing with Python

[Sturla Molden]

On 2/19/2009 3:34 AM, Adam Olsen wrote:

> * Erlang's modules/classes/functions are not first-class objects, so
> it doesn't need a copy of them.  Python does, so each interpreter
> would have a memory footprint about the same as a true process.

Yes, each interpreter will have the memory footprint of an interpreter. 
So the memory use would be about the same as with multiprocessing.

> * Any communication requires a serialize/copy/deserialize sequence.

No it does not, and this why embedded interpreters are better than 
multiple processes (cf. multiprocessing). Since the interpreters share 
virtual memory, many objects can be shared without any serialization.
That is, C pointers will be valid in both interpreters, so it should in 
many cases be possible to pass a PyObject* from one interpreter to 
another. This kind of communication would be easiest to achieve with 
immutable objects.

Another advantage is that there will be just one process to kill, 
whenever that is required.



S.M.