extension module performance problems

[rwgk]

I wrote a tiny shared library (.so) extension module for
tokenizing input files. Unfortunately this implementation
ran only marginally faster than a pure-Python solution.
I did some profiling on a Silicon Graphics (IRIX 6.5) and
noticed that significant time is spent in the pthreads library
although there is only one thread. Therefore I recompiled
python (1.5.2) without threading, and the program ran about
three times faster.

In contrast, the runtime on a Compaq Alpha (Tru64 5.0) is not
affected by the inclusion of pthreads. However, my test
runs about four times faster if the extension module is
linked statically.

Here is the summary of CPU times:

         no threads    with threads
        static   .so   static   .so
Alpha    0.6    2.4     2.4    2.4
SGI      0.83   0.83    2.19   2.25

Can these observations be explained? Is there a way to
get full performance under all circumstances?

The python bit of the test program is attached.

Thanks!
        Ralf


import sys
import _cnsinput

if (len(sys.argv) < 2):
  f = sys.stdin
else:
  f = open(sys.argv[1], 'r')

try:
  io = _cnsinput.start(f)
  gNW = io.getNextWord
  while 1:
    word = gNW() # this line is executed 69768 times
except EOFError:
  print 'end of file'

f.close()





-----------------------------------------------------------

Got questions?  Get answers over the phone at Keen.com.
Up to 100 minutes free!
http://www.keen.com