Python threading (was: Re: global interpreter lock not working as it should)

[anton wilson]

On Saturday 03 August 2002 04:56 am, Jonathan Hogg wrote:
> On 3/8/2002 2:43, in article aifcfh$jha$0 at 216.39.172.122, "Bengt Richter"
>
> <bokr at oz.net> wrote:
> >> This is true. Therefore, the only time another thread WILL grab the GIL
> >> under Linux is if
> >>
> >> 1) the GIL is released by the currently running thread
> >> 2) the thread that just released the GIL depletes its timeslice before
> >> it can grab the lock again
> >> 3) the OS notices the process has depleted it's timeslice and the yanks
> >>   it from the CPU (this happens every 100 times per second by default on
> >> an i386)
> >> 4) the waiting thread that recieved the GIL release signal is chosen to
> >> run
> >>
> >> Therefore, we now have a large set of coincidences for CPU-bound python
> >> threads. The only reason it works at all is because it happens 100 times
> >> per second and the GIL is released frequently by default. So there is a
> >> sufficient probability that these 4 cases will happen simultaneously.
> >
> > Hm. If that's an accurate description (and I am skeptical ;-), I don't
> > think I like it.
> > Seems like if another thread is waiting for the GIL, then after a full
> > time slice the
> > current GIL-holder ought to be told to give it up ASAP and be given a new
> > time slice
> > only to do the minimum it needs to do that.
>
> If this is true, and I'm also skeptical, then it's a bug in Linux not
> Python and Python should do nothing about it.
>
> I ran a little experiment (code enclosed below) to crudely check the thread
> switching on my machine (Mac OS X - well known for it's threading). Running
> 10 CPU-bound threads together for 10 seconds came up with them all getting
> fairly even CPU time (within +/- 10% of the mean).
>
> My crude count for the actual number of thread switches that occurred
> during those 10 seconds showed never less than 1200 thread switches per
> second and up to 3000 thread switches per second.
>
> I also tried the same code on a FreeBSD, Solaris, and Linux box. The
> summary results were:
>
>     OS               Architecture   Python     Thread Switches (avg 2 runs)
>     Mac OS X 10.1    PPC            2.3-CVS    24971
>     FreeBSD 4.3      i386           2.2.1      158499
>     Linux 2.2.14     i386           2.2        18281
>     Solaris 8        SPARC          2.2.1      243
>
> So an interesting spread with Solaris being way down there in terms of how
> often it switched threads (though the CPU distribution seemed to be pretty
> fair anyway) - I'm not sure what threading library Python was compiled
> against on the Solaris box (which has two - one OS and one user), I might
> look into that out of curiosity. Anyone know FreeBSD well enough to comment
> on the results for that platform?
>
> Note that Linux is still doing over 1800 switches per second, which makes
> me even more skeptical that it is only capable of thread switching due to a
> huge coincidence.


Well, when I said coincidence, I didn't necessarily mean huge, it works, but 
when compared to the number of times it tries to give up the interpreter . . 
. 

The timer tick does happen 100x per second and since python gives up the lock 
every 10 us or so, it's really not a huge coincidence for an overlap. I just 
don't really like the reliance on this coincidence.

>
> Any comments?
>
> Jonathan
>
>
> ----- threads.py -----
> import threading
> import time
> import operator
>
> def sum( xs ):
>     return reduce( operator.add, xs )
>
> QUITTING = 0
> RUNNING = None
>
> class Counter( threading.Thread ):
>     def run( self ):
>         global QUITTING, RUNNING
>         count = 0
>         switches = 0
>         while not QUITTING:
>             count += 1
>             if self is not RUNNING:
>                 RUNNING = self
>                 switches += 1
>         self.count = count
>         self.switches = switches
>
> counters = [ Counter() for i in range(10) ]
>
> for counter in counters:
>     counter.start()
>
> time.sleep( 10.0 )
>
> QUITTING = 1
> for counter in counters:
>     counter.join()
>
> counts = [ counter.count for counter in counters ]
> switches = [ counter.switches for counter in counters ]
>
> print "Counts:"
> print counts
> print "Total =", sum( counts )
>
> print "\nSwitches:"
> print switches
> print "Total =", sum( switches )