Processing a file using multithreads
abhishek.vit at gmail.com
Thu Sep 8 18:49:51 EDT 2011
My experience with python is 2 days and I am looking for a slick way
to use multi-threading to process a file. Here is what I would like to
do which is somewhat similar to MapReduce in concept.
# test case
1. My input file is 10 GB.
2. I want to open 10 file handles each handling 1 GB of the file
3. Each file handle is processed in by an individual thread using the
same function ( so total 10 cores are assumed to be available on the
4. There will be 10 different output files
5. once the 10 jobs are complete a reduce kind of function will
combine the output.
Could you give some ideas ?
So given a file I would like to read it in #N chunks through #N file
handles and process each of them separately.
More information about the Python-list