Problem in threading

[Steve Holden]

Gurpreet Sachdeva wrote:

>>>>So That means blindly using threads on any process won't help!
>>
>>It depends on what "help" means to you.
> 
> 
> Help means to improve processing speed in achieving a particular
> task... *Help* here also means that I have a processor farm, how do I
> best use them to get maximum processing speed out of them...
> 
And that's the crux of your problem.

Firstly, you've discovered that attempts to partition a task using 
threads won't work well with a compute-intensive algorithm, since 
multiple threads will simply contend against each other for CPU in a 
single process.

This is not assisted by Python's use of a global interpreter lock (GIL) 
to ensure thread-safety.

A thread can release the GIL, but typically it will do this only when 
it's involved in some blocking operation. This means that threading can 
be useful to speed up network operations, for example, but even then you 
might get a better speedup using explicitly asynchronous techniques 
based on non-blocking sockets.
> 
>>The example you listed isn't a good use of thread for performance
> 
> 
> The example was a dummy one to become friendly with Threads and to
> understand the working/power of them... The example which will be
> finally used with threads take 5 Days to complete... I want to convert
> that in few hours!
> 
In that case you definitely need to be looking at multiprocess 
algorithms if you are sticking with Python. Since each process has its 
own copy of the interpreter, they each also have their own GIL, and so 
the operating system will be able to schedule the processes in parallel 
on separate CPUs.

> Long Journey ahead...
> 
Indeed, but an interesting one, no doubt. Good luck.

regards
  Steve
-- 
Steve Holden               http://www.holdenweb.com/
Python Web Programming  http://pydish.holdenweb.com/
Holden Web LLC      +1 703 861 4237  +1 800 494 3119