Embedding Python, threading and scalability

[Andrew Dalke]

Wenning Qiu:
> I am researching issues related to emdedding Python in C++ for a
> project.

> Has anyone on this list run into the same problem that I have, or does
> anyone know of any plan of totally insulating multiple embedded Python
> interpreters?

Ahh, the Global Interpreter Lock (GIL).

Years ago, Greg Stein had a version of Python 1.4 running with no GIL.

http://www.python.org/ftp/python/contrib-09-Dec-1999/System/threading.README
Search for "free threading" to get more hits on this topic.

As I recalled, it slowed down the performance on
single-processor/single-threaded
machines, so the general take was to keep the GIL.  In addition, see
http://groups.google.com/groups?selm=mailman.1008992607.2279.python-list%40p
ython.org&oe=UTF-8&output=gplain
Tim Peters:
] The prospects for another version of that grow dimmer.  Everyone (incl.
] Greg) has noticed that CPython internals, over time, increase their
reliance
] on the thread-safety guarantees of the global interpreter lock.

The only solutions I know of are explicit multi-process solutions:
  - a generic system, like XML-RPC/SOAP/PVM/MPI/CORBA, on
       which you build your own messaging system
  - use systems like Pyro or Twisted, which understand Python objects
       and implement 'transparent' proxying via network communications
  - use POSH, which does the proxying through shared memory (but
       this uses Intel-specific assembly)

                    Andrew
                    dalke at dalkescientific.com