On Wed, Mar 13, 2013 at 9:54 AM, Andrea Cimatoribus <Andrea.Cimatoribus@nioz.nl> wrote:

Thanks a lot for the feedback, I'll try to modify my function to overcome this issue.
Since I'm in the process of buying new hardware too, a slight OT (but definitely related).
Does an ssd provide substantial improvement in these cases?

It should. Seek time on an ssd is quite low, and readout is fast. Skipping over items will probably not be as fast as a sequential read but I expect it will be substantially faster than a disk. Nathaniel's loop idea will probably work faster also. The sequential readout rate of a modern ssd will be about 500 MB/sec, so you can probably just divide that into your file size to get an estimate of the time needed.

<snip>

Chuck