Processing a file using multithreads

Gregory Ewing greg.ewing at canterbury.ac.nz
Fri Sep 9 02:03:45 CEST 2011


Abhishek Pratap wrote:

> 3. Each file handle is processed in by an individual thread using the
> same function ( so total 10 cores are assumed to be available on the
> machine)

Are you expecting the processing to be CPU bound or
I/O bound?

If it's I/O bound, multiple cores won't help you, and
neither will threading, because it's the disk doing the
work, not the CPU.

If it's CPU bound, multiple threads in one Python process
won't help, because of the GIL. You'll have to fork
multiple OS processes in order to get Python code running
in parallel on different cores.

-- 
Greg



More information about the Python-list mailing list