global interpreter lock not working as it should

[Tim Peters]

[anton wilson]
> ...
> What's the purpose of releasing an reaquiring the lock if no
> other threads can run?

They can, as other responses have amply demonstrated.  Whether they *will*
is up to the platform thread package, which was presumably designed to pass
out timeslices in a way maximally efficient for it.  It's not Python's job
to force thread switches against the better judgment of the thread facility
authors; you can do that yourself if you think it's required, e.g. via
time.sleep(0.001).

A peculiar thing here is that "the Python lock" is an abstraction.  You said
you were running on Linux, and "the Python lock" isn't a lock there.  In
Pythons released to date, a Python lock under pthreads is a combination of a
phtreads mutex never held across more than a few C instructions, and a
pthreads condition variable.  When "the lock" is released, a condvar signal
is done to notify some other thread that it can grab the lock.  This happens
before the relinquishing thread tries to reacquire the lock.  So the
implementation of condition variables on your box apparently decides not to
take action on a signal at once, dallying until the original thread has
already reacquired "the lock".  That's fine -- it's allowed to do that, and
it surely has its own performance reasons for doing so.

If you try the current CVS version of Python, the implementation of "the
lock" on Linux has changed to use POSIX semaphores instead.  They may or may
not act more like you hope -- I don't know (haven't tried it).  It seems
possible that they might, since semaphores came out of the POSIX realtime
extensions, where people feel much freer to make unreasonable <wink>
demands.

> ...
> I have a feeling that there is a hidden race condition between when the
> thread wakes up the other thread and tries to reaquire the lock.

"The race" would be in the platform implementation of pthread_cond_signal(),
but I expect that's working the way its author intended.