Multi-threading python

[Michael Chermside]

David Brown wrote:
> I've just been reading a little about the GIL and threading in python, and I
> have a couple of questions that I hope someone can answer.

I'm not an expert, but I'll try.

> Am I right in thinking that threading in Python does not use the underlying
> OS's threads, but effectively implements its own scheduler?  Or does it use
> the OS's threads, but uses the GIL to ensure that only one is runable at a
> time?

It uses the OS's threads, and then uses the GIL to ensure that only one 
is executing Python bytecodes at a time. However, other threads may be 
runable... they could be blocked waiting on I/O, or executing some C 
code from a module that releases the GIL before performing a 
long-running computation.

> On a dual-processor system, does this mean that only one thread of a Python
> program can run at a time, even when both CPUs are free, so that two
> seperate python interpreter processes are needed to take advantage of the
> two CPUs?

Yes.

> What about when a Python thread is blocked for some other reason, such as
> while waiting for a file read - will that block other threads, or will the
> waiting thread definitely release the GIL and allow other threads to run?

Depends on what it's blocked for. If it's blocked for I/O, then the GIL 
is released and the other threads will run. I'm not sure what else it 
would block for that might be a problem. But you could certainly CREATE 
a problem if you tried... simplest way would be to write C code which 
ran for a long time WITHOUT releasing the GIL.

It would be much more difficult to block things through badly written 
Python code, since after each bytecode the interpretor can switch to a 
different thread.

> If the GIL does block all threads in this manner, are there any plans to
> introduce more fine-grained locking to improve scalability, analagous to
> locking in the Linux kernel?

No. The general consensus is that eliminating the GIL would be quite 
difficult and that there's not really all that much to gain. (Most 
casual users don't have multi-processor systems; serious users of this 
sort probably use Numeric and/or put the tight loops into C anyhow.) 
However, if you know someone who's good with this stuff and wants to 
volunteer, I'm sure they wouldn't turn down working code (unless it 
introduced other problems).

-- Michael Chermside