Processing a file using multithreads
greg.ewing at canterbury.ac.nz
Fri Sep 9 02:03:45 CEST 2011
Abhishek Pratap wrote:
> 3. Each file handle is processed in by an individual thread using the
> same function ( so total 10 cores are assumed to be available on the
Are you expecting the processing to be CPU bound or
If it's I/O bound, multiple cores won't help you, and
neither will threading, because it's the disk doing the
work, not the CPU.
If it's CPU bound, multiple threads in one Python process
won't help, because of the GIL. You'll have to fork
multiple OS processes in order to get Python code running
in parallel on different cores.
More information about the Python-list