Problem in threading
steve at holdenweb.com
Thu Dec 30 14:21:18 CET 2004
Gurpreet Sachdeva wrote:
>>>>So That means blindly using threads on any process won't help!
>>It depends on what "help" means to you.
> Help means to improve processing speed in achieving a particular
> task... *Help* here also means that I have a processor farm, how do I
> best use them to get maximum processing speed out of them...
And that's the crux of your problem.
Firstly, you've discovered that attempts to partition a task using
threads won't work well with a compute-intensive algorithm, since
multiple threads will simply contend against each other for CPU in a
This is not assisted by Python's use of a global interpreter lock (GIL)
to ensure thread-safety.
A thread can release the GIL, but typically it will do this only when
it's involved in some blocking operation. This means that threading can
be useful to speed up network operations, for example, but even then you
might get a better speedup using explicitly asynchronous techniques
based on non-blocking sockets.
>>The example you listed isn't a good use of thread for performance
> The example was a dummy one to become friendly with Threads and to
> understand the working/power of them... The example which will be
> finally used with threads take 5 Days to complete... I want to convert
> that in few hours!
In that case you definitely need to be looking at multiprocess
algorithms if you are sticking with Python. Since each process has its
own copy of the interpreter, they each also have their own GIL, and so
the operating system will be able to schedule the processes in parallel
on separate CPUs.
> Long Journey ahead...
Indeed, but an interesting one, no doubt. Good luck.
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119
More information about the Python-list