Exploiting Dual Core's with Py_NewInterpreter's separated GIL ?

[Shane Hathaway]

robert wrote:
> I'd like to use multiple CPU cores for selected time consuming Python
> computations (incl. numpy/scipy) in a frictionless manner.
> 
> Interprocess communication is tedious and out of question, so I
> thought about simply using a more Python interpreter instances
> (Py_NewInterpreter) with extra GIL in the same process. I expect to
> be able to directly push around Python Object-Trees between the 2 (or
> more) interpreters by doing some careful locking.
> 
> Any hope to come through? If possible, what are the main dangers? Is
> there an example / guideline around for that task? - using ctypes or
> so.
> 
> Or is there even a ready made Python module which makes it easy to
> setup and deal with extra Interpreter instances? If not, would it be
> an idea to create such thing in the Python std libs to make Python
> multi-processor-ready. I guess Python will always have a GIL -
> otherwise it would loose lots of comfort in threaded programming

I'd like to mention mod_python, which creates multiple interpreters
inside Apache.  It does this transparently and works well.  There is no
apparent lock contention between the interpreters, because no Python
objects are shared.

Also, I'd like to make the observation that it is not always necessary
to share the entire interpreter (ala threads) in order to take advantage
of multiple cores.  I think Python only needs a nice way to share a
relatively small set of objects using shared memory.  POSH goes in that
direction, but I don't think it's simple enough yet.

http://modpython.org/
http://poshmodule.sourceforge.net/

Shane