[Python-ideas] Parallel processing with Python

[Adam Olsen]

On Thu, Feb 19, 2009 at 3:53 AM, Sturla Molden <sturla at molden.no> wrote:
> On 2/19/2009 3:34 AM, Adam Olsen wrote:
>> * Any communication requires a serialize/copy/deserialize sequence.
>
> No it does not, and this why embedded interpreters are better than multiple
> processes (cf. multiprocessing). Since the interpreters share virtual
> memory, many objects can be shared without any serialization.
> That is, C pointers will be valid in both interpreters, so it should in many
> cases be possible to pass a PyObject* from one interpreter to another. This
> kind of communication would be easiest to achieve with immutable objects.

If you could directly use another interpreter's PyObject in the
current interpreter then they wouldn't separate interpreters.  You
need to apply it for the type objects too, and if you start sharing
those you'll kill any performance advantage of this whole scheme.

The realistic scenario is you treat each interpreter as a monitor: you
can call into another interpreter quite cheaply (release your GIL, set
your current interpreter to be them, acquire their GIL).  However,
since you are only in one at any given point in time, you need to copy
anything you want to transmit.

To put it another way, your proxy objects can hold pointers to the
other interpreter's objects, but you can't use them until you go back
into that other interpreter.


-- 
Adam Olsen, aka Rhamphoryncus