Embedding Python, threading and scalability

[Afanasiy]

On 8 Jul 2003 14:54:22 -0700, wenning_qiu at csgsystems.com (Wenning Qiu)
wrote:

>I am researching issues related to emdedding Python in C++ for a
>project.
>
>My project will be running on an SMP box and requires scalability.
>However, my test shows that Python threading has very poor performance
>in terms of  scaling. In fact it doesn't scale at all.
>
>I wrote a simple test program to complete given number of iterations
>of a simple loop. The total number of iterations can be divided evenly
>among a number of threads. My test shows that as the number of threads
>grows, the CPU usage grows and the response time gets longer. For
>example, to complete the same amount of work, one thread takes 10
>seconds, 2 threads take 20 seconds and 3 threads take 30 seconds.
>
>The fundamental reason for lacking scalability is that Python uses a
>global interpreter lock for thread safety. That global lock must be
>held by a thread before it can safely access Python objects.

I asked once and was told it was best fixed by removing the documentation
which mentioned it. Others also stated it was unlikely to be fixed.

http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&threadm=53u1evk5jmdcgma5e8eupbe3tn45js302i%404ax.com&rnum=1&prev=/groups%3Fhl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3DUTF-8%26selm%3D53u1evk5jmdcgma5e8eupbe3tn45js302i%25404ax.com

However, I believe Lua since 4-work4, just before Lua 5, solved this.
Unfortunately Lua is not Python.

Another thing to consider if you care about SMP, is your C/C++ memory
management, assuming you aren't using something custom already, maybe a
shared heap. I have worked wonders with libhoard (and SmartHeap,
commercially). Some applications will run slower on SMP than if you
removed one of the processors.

www.hoard.org
www.microquill.com

mmm, graphs

-AB