[Tutor] Multi-threading IO and the GIL

[Kent Johnson]

On Tue, Oct 28, 2008 at 7:27 AM, Tim Kelsey <tk at eutechnyx.com> wrote:

> Im not sure if this is the correct list to post this sort of thing

We'll try to answer pretty much any Python question.

> but I
> have a question about python threading and the GIL. I have a multi-threaded
> application that is hitting a MySQL database for some info from within the
> various threads. When increase the size of my thread pool I see little or no
> improvement in the application performance but the CPU usage is not maxed.
> This suggests to me that the app is IO bound on the database, surly more
> threads should help in this case.

I can think of a few other possibilities...

Does the app actually create more threads? If you only create two
threads, the size of the thread pool doesn't much matter.

It's possible that all the threads are blocking on the database, for
example if they are all trying to modify the same row. Or the app
could truly be I/O bound, if it is reading large amounts of data from
the DB then more threads trying to read more data from the same source
may not help.

> Is it possible that the GIL is not being release by a thread during the time
> its blocking on the DB connection? Im using MySQLdb module and the
> application is purely written in python.

I don't know but it seems unlikely. Is there a MySQLdb list where you could ask?

Kent