Threads + Re(gex) Baffled!?!?!?

[Tim Peters]

[Cliff Daniel, catalogs random symptoms running a threaded program
 under UE5000 with 8 cpus, Solaris 2.6, Python 1.5.2]

You didn't mention which flavor of threads.  Over the years many have
reported better luck running pthreads (as opposed to native threads) under
Solaris; OTOH, reports of the opposite aren't uncommon either <wink>.

> ...
> For practical purposes I have spawned 15 worker threads who
> each will connect to a different host (480 Hosts total).  They would
> then execute code similar to this:
>
> ---------------------------------------
> Sock.send('some-command')
> Reply = ReadUntil('> ', Sock, Timeout)
>
> if Reply == '':
>     return(-1)
> else:
>     tmp = re.split('\r\n', Reply)
>     for line in tmp:
> 	m = pat.match(line)
> 	.etc.

If that code is really at top level, Reply, tmp, line and m are shared vrbls
and the code isn't thread-safe.  That's a problem with looking at program
fragments -- we can't tell what's important and what's an artifact of your
attempt to summarize.  If you need significant help with this, I'm afraid
you're going to have to post a self-contained, minimal, executable app (but
to the Thread-SIG, not here).

> Upon running the script you see nice and speedy results at first with
> very little cpu usage.  However, the cpu usage grows continously until
> the program ends.  I have noticed that when the bloating occurs the
> a lot of the cpu time is spent in the kernel which would lead me to
> believe this is some sort of locking issue hidden within the 're'
> module?  When I put a 'continue' in front of the pat.match() every
> thing works just fine.

Two things odd about that:

1) You're still calling re ("tmp = re.split(...").  (BTW, use string.split
for this -- faster and clearer)

2) re knows nothing about threads; there are no locking, or any other
thread-aware, operations in it.  It seems to have been coded to be
thread-safe, but Python isn't exploiting that at present.  So wrt threads
it's exactly like any other random chunk of thread-unaware C code in the
implementation:  it runs single-threaded by virtue of Python's global
interpreter lock.  So if you're having a problem with re, you'll most likely
have the same problem with any piece of C code.

> The pattern I'm matching is as follows:
> pat = re.compile('\s+(\w+)\s+\(.+?\):\s+(\d+)\s+(\d+)')

That looks efficient provided that most lines match the pattern.  If they
don't, it may trash some trying all possible ways to (fail to) match the
ambiguous sub-expression

    .+?\)

(which will match any number of right parens in an ultimately futile attempt
to let the rest of the pattern match).  If what you really want to do is
match up to the leftmost right paren,

    [^}]+\)

expresses that directly (and also more efficiently),

> Is there any known issues with locking and 're'?

No, and best I can tell it's impossible for there to be a locking issue
unique to re.

> If anyone has any ideas about this I desperately need some help.  I
> hate to have to dump this entire project for performance reasons.

The bad news is that, so far, there's nothing here to suggest you wouldn't
have exactly the same symptoms running single-threaded straight C code.  We
need more info before suspecting Python is compelling, or you need to find a
wizard for your flavor of Solaris threads.

the-good-news-is-that-they're-a-dime-a-dozen<wink>-ly y'rs  - tim