Python threading (was: Re: global interpreter lock not working as it should)

[Andrew MacIntyre]

On Sat, 3 Aug 2002, Jonathan Hogg wrote:

> > bash-2.04$ python threads.py
> > Counts:
> > [364324, 154922, 14009, 14005, 14001, 13996, 13993, 13989, 13985, 13981]
> > Total = 631205
>
> This on the other hand is very interesting. The first couple of threads are
> clearly dominating the CPU. I went back and ran threads.py again on my
> FreeBSD box and noticed similar behaviour that I had overlooked before:
>
> orwell% python threads.py
> Counts:
> [54770, 55971, 30312, 15698, 15694, 15661, 15634, 15612, 15593, 15575]
> Total = 250520
>
> Switches:
> [15702, 15704, 15701, 15696, 15693, 15661, 15634, 15612, 15593, 15575]
> Total = 156571
>
> I presume this means FreeBSD switches to the first thread immediately after
> it is started. This thread then gets to run at full tilt for a period of
> time before it switches back to the main thread, that starts another thread
> and then the two of them must share the CPU for a while, then the main
> thread starts a third etc.
>
> The other OSen I tested on didn't exhibit this behaviour so I guess they
> must mark the threads as being runnable but not actually re-schedule until
> the main thread hits the sleep.
>
> Still can't figure out why FreeBSD is doing so much thread switching though.
> It must be pretty inefficient.

I've finally (very belatedly) gotten to playing with this...

I've just run CVS Python on my FreeBSD 4.4 box with the following results:
- Python version:

  Python 2.3a0 (#1, Aug  8 2002, 11:24:34)
  [GCC 2.95.3 20010315 (release) [FreeBSD]] on freebsd4

- RobinB's tspeed.py:

  $ ./python tspeed.py 10000000
  n=10000000 s=10000000 t=126.935

- your threads.py

  Counts:
  [141459, 172396, 88760, 81967, 102945, 114991, 70182, 80271, 68742,
    45263]
  Total = 966976

  Switches:
  [22, 21, 21, 19, 19, 17, 15, 14, 13, 12]
  Total = 173


Python 2.1.1 on same system:
- version:

  Python 2.1.1 (#1, Sep 13 2001, 18:12:15)
  [GCC 2.95.3 20010315 (release) [FreeBSD]] on freebsd4

- RobinB's tspeed.py:

  $ python tspeed.py 10000000
  n=10000000 s=10000000 t=139.876

- your threads.py

  Counts:
  [18259, 7118, 7276, 4943, 4931, 4923, 4915, 4909, 4902, 4897]
  Total = 67073

  Switches:
  [4952, 4952, 4950, 4943, 4931, 4923, 4915, 4909, 4902, 4897]
  Total = 49274


CVS Python on OS/2 EMX:
- Python version:

  Python 2.3a0 (#0, Aug  8 2002, 22:21:58) [EMX GCC 2.8.1] on os2emx

- RobinB's tspeed.py:

  F:\dev\cvs\python-cvs\PC\os2emx>python tspeed.py 10000000
  n=10000000 s=10000000 t=12.560

- your threads.py

  Counts:
  [1935007, 440530, 219118, 1826061, 78, 216695, 1087462, 70, 15185927,
    64]
  Total = 20911012

  Switches:
  [91, 91, 87, 85, 77, 77, 77, 69, 70, 63]
  Total = 787


I believe that others have noted that the implementation of the GIL
changes with 2.3 on pthreads - I've forgotten the details, but its clear
that FreeBSD supports the 2.3 alternative (which is used when available,
otherwise the pre-2.3 implementation is used).

OS/2 threading is relatively stingy with thread switches too... (the EMX
port shares the OS/2 thread implementation with the VACPP port).  I
haven't tried tuning the thread settings on my OS/2 box, so I've no idea
how those numbers might be able to be tweaked.  Several runs of threads.py
nearly always produced 2 threads with counts < 100, and 1 or 2 threads
with a total count of ~15000000, but always a wild variation in the
counts.

--
Andrew I MacIntyre                     "These thoughts are mine alone..."
E-mail: andymac at bullseye.apana.org.au  | Snail: PO Box 370
        andymac at pcug.org.au            |        Belconnen  ACT  2616
Web:    http://www.andymac.org/        |        Australia