Andrea Gavana wrote:
I have tried the solutions proposed in the previous thread and it looks like Chris' one is the fastest for my purposes.
whoo hoo! What do I win? ;-)
Splitting the reading process between 4 processes will require the exchange of 5-20 MB from the child processes to the main one: do you think my script will benefit from using multiprocessing?
If you are talking about multiprocessing to read the data in -- I don't think so -- that's probably IO bound anyway. You can't make your disks faster with multiple processors.
Should I try another approach?
I don't know it will do anything for performance, but you might want to look at memory mapped arrays -- it's a very cool way to work with data files too big to want to bring into memory all at once. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov