Python threading (was: Re: global interpreter lock not working as it should)

[Armin Steinhoff]

Jonathan Hogg <jonathan at onegoodidea.com> wrote in message news:<B9715946.F0A5%jonathan at onegoodidea.com>...
> On 3/8/2002 2:43, in article aifcfh$jha$0 at 216.39.172.122, "Bengt Richter"
> <bokr at oz.net> wrote:
> 

> 
> I also tried the same code on a FreeBSD, Solaris, and Linux box. The summary
> results were:
> 
>     OS               Architecture   Python     Thread Switches (avg 2 runs)
>     Mac OS X 10.1    PPC            2.3-CVS    24971
>     FreeBSD 4.3      i386           2.2.1      158499
>     Linux 2.2.14     i386           2.2        18281
>     Solaris 8        SPARC          2.2.1      243
> 
> 
Hi All,


I have build three versions of python by inserting a sched_yield and a
delay of
1ms in the code of ceval.c below ... and did run Jonathans testcode.

>From cevel.c ...
#ifdef WITH_THREAD
			if (interpreter_lock) {
				/* Give another thread a chance */

				if (PyThreadState_Swap(NULL) != tstate)
					Py_FatalError("ceval: tstate mix-up");
				PyThread_release_lock(interpreter_lock);

				/* Other threads may run now */   case: unmodified
            sched_yield()                     case: sched_yield
            delay(1)                          case: delay

				PyThread_acquire_lock(interpreter_lock, 1);
				if (PyThreadState_Swap(tstate) != NULL)
					Py_FatalError("ceval: orphan tstate");
			}
#endif

Here are the results:

case ceval.c unmodified:

>>> execfile('/root/threads.py')
Counts:
[207251, 189529, 228940, 203701, 216320, 169515, 218877, 223871,
185256, 212550]
Total = 2055810

Switches:
[85, 85, 83, 84, 84, 84, 84, 83, 84, 86]
Total = 842
>>>

That means the timeslice exhausted 85 times (or jumped to a higher
priority) at 'other threads may run now' during 207251 loops! (More
often than I could imagine ... )

This leads to an awful bad thread switching performance!


case sched_yield:

Counts:
[110635, 110589, 110598, 110597, 110617, 110585, 110600, 110597,
110604, 110587]
Total = 1106009

Switches:
[110584, 110549, 110559, 110558, 110589, 110549, 110561, 110553,
110562, 110548]
Total = 1105612
>>>

This says it all ....


case delay:

Counts:
[1438, 1434, 1431, 1428, 1425, 1422, 1419, 1416, 1412, 1409]
Total = 14234

Switches:
[1434, 1434, 1431, 1428, 1425, 1422, 1419, 1416, 1412, 1409]
Total = 14230

We have a slow interpreter ... but a good thread switching performance
:-)

-------------------------------------------------------------------------------
My conclusion: insert a sched_yield if the ceval.c code is called from
a thread.

#ifdef WITH_THREAD
			if (interpreter_lock) {
				/* Give another thread a chance */

				if (PyThreadState_Swap(NULL) != tstate)
					Py_FatalError("ceval: tstate mix-up");
				PyThread_release_lock(interpreter_lock);

				/* Other threads may run now */ 
-->          if (called_from_a_thread)
-->              sched_yield()               

				PyThread_acquire_lock(interpreter_lock, 1);
				if (PyThreadState_Swap(tstate) != NULL)
					Py_FatalError("ceval: orphan tstate");
			}
#endif

Armin