[Tutor] concurrent file reading using python
abhishek.vit at gmail.com
Mon Mar 26 20:05:09 CEST 2012
I want to utilize the power of cores on my server and read big files
(> 50Gb) simultaneously by seeking to N locations. Process each
separate chunk and merge the output. Very similar to MapReduce
What I want to know is the best way to read a file concurrently. I
have read about file-handle.seek(), os.lseek() but not sure if thats
the way to go. Any used cases would be of help.
PS: did find some links on stackoverflow but it was not clear to me if
I found the right solution.
More information about the Tutor