On 2/19/2009 3:34 AM, Adam Olsen wrote:

* Erlang's modules/classes/functions are not first-class objects, so it doesn't need a copy of them. Python does, so each interpreter would have a memory footprint about the same as a true process.

Yes, each interpreter will have the memory footprint of an interpreter. So the memory use would be about the same as with multiprocessing.

* Any communication requires a serialize/copy/deserialize sequence.

No it does not, and this why embedded interpreters are better than multiple processes (cf. multiprocessing). Since the interpreters share virtual memory, many objects can be shared without any serialization. That is, C pointers will be valid in both interpreters, so it should in many cases be possible to pass a PyObject* from one interpreter to another. This kind of communication would be easiest to achieve with immutable objects. Another advantage is that there will be just one process to kill, whenever that is required. S.M.

On Thu, Feb 19, 2009 at 2:53 AM, Sturla Molden wrote:

Only if you have an approach to GC that does not require locking. The current reference counting macros are not thread-safe so every INCREF and DECREF would have to be somehow protected by a lock or turned into an atomic operation. Recall that many frequently used objects, like None, small integers, and interned strings for common identifiers, are shared and constantly INCREFed and DECREFed.

Thanks for the info. I think this would be just a minor inconvinience. Sending a message in the form of PyObject *x from A to B could perhaps be solved like this: Interpreter A: Increfs immutable pyobj x aquires the GIL of interpreter B messages pyobject x to interpreter B releases the GIL of interpreter B Interpreter B: Creates a proxy object p for reading attributes of x Increfs & decrefs p (refcounts for x or its attributes are not touched by p) when p is collected: aquires the GIL of interpreter A decrefs x releases the GIL of interpreter A Synchronization of reference counting is obviously needed (and the GIL is great for that). But serialization of the whole object would be avoided. This would depend on immutability of the message object. S.M.

On Thu, Feb 19, 2009 at 3:53 AM, Sturla Molden wrote:

On 2/19/2009 3:34 AM, Adam Olsen wrote:

...

* Any communication requires a serialize/copy/deserialize sequence.

No it does not, and this why embedded interpreters are better than multiple processes (cf. multiprocessing). Since the interpreters share virtual memory, many objects can be shared without any serialization. That is, C pointers will be valid in both interpreters, so it should in many cases be possible to pass a PyObject* from one interpreter to another. This kind of communication would be easiest to achieve with immutable objects.

If you could directly use another interpreter's PyObject in the current interpreter then they wouldn't separate interpreters. You need to apply it for the type objects too, and if you start sharing those you'll kill any performance advantage of this whole scheme. The realistic scenario is you treat each interpreter as a monitor: you can call into another interpreter quite cheaply (release your GIL, set your current interpreter to be them, acquire their GIL). However, since you are only in one at any given point in time, you need to copy anything you want to transmit. To put it another way, your proxy objects can hold pointers to the other interpreter's objects, but you can't use them until you go back into that other interpreter. -- Adam Olsen, aka Rhamphoryncus