Embedding Python, threading and scalability
abelikov72 at hotmail.com
Thu Jul 10 03:14:06 CEST 2003
On 8 Jul 2003 14:54:22 -0700, wenning_qiu at csgsystems.com (Wenning Qiu)
>I am researching issues related to emdedding Python in C++ for a
>My project will be running on an SMP box and requires scalability.
>However, my test shows that Python threading has very poor performance
>in terms of scaling. In fact it doesn't scale at all.
>I wrote a simple test program to complete given number of iterations
>of a simple loop. The total number of iterations can be divided evenly
>among a number of threads. My test shows that as the number of threads
>grows, the CPU usage grows and the response time gets longer. For
>example, to complete the same amount of work, one thread takes 10
>seconds, 2 threads take 20 seconds and 3 threads take 30 seconds.
>The fundamental reason for lacking scalability is that Python uses a
>global interpreter lock for thread safety. That global lock must be
>held by a thread before it can safely access Python objects.
I asked once and was told it was best fixed by removing the documentation
which mentioned it. Others also stated it was unlikely to be fixed.
However, I believe Lua since 4-work4, just before Lua 5, solved this.
Unfortunately Lua is not Python.
Another thing to consider if you care about SMP, is your C/C++ memory
management, assuming you aren't using something custom already, maybe a
shared heap. I have worked wonders with libhoard (and SmartHeap,
commercially). Some applications will run slower on SMP than if you
removed one of the processors.
More information about the Python-list