Embedding Python, threading and scalability

[Aahz]

In article <ec23a1ae.0307081354.5fc06cb at posting.google.com>,
Wenning Qiu <wenning_qiu at csgsystems.com> wrote:
>
>My project will be running on an SMP box and requires scalability.
>However, my test shows that Python threading has very poor performance
>in terms of  scaling. In fact it doesn't scale at all.

That's true for pure Python code.

>The fundamental reason for lacking scalability is that Python uses a
>global interpreter lock for thread safety. That global lock must be
>held by a thread before it can safely access Python objects.

Correct.  The problem is that the GIL makes Python more efficient in
many ways, because there's no need for fine-grained locking.  You're
using Python inside-out for this purpose -- the way to scale Python in a
threaded environment is to call out to a C extension that releases the
GIL.

>Has anyone on this list run into the same problem that I have, or does
>anyone know of any plan of totally insulating multiple embedded Python
>interpreters?

Sure!  Use multiple processes.

Other people have mentioned Perl and Tcl in this thread.  I wonder how
they deal with the problem of loading DLLs with static data.
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"Not everything in life has a clue in front of it...."  --JMS