[Tutor] Load Entire File into memory

[Steven D'Aprano]

On Mon, Nov 04, 2013 at 06:02:47PM -0800, Danny Yoo wrote:

> To visualize the sheer scale of the problem, see:
> 
>     http://i.imgur.com/X1Hi1.gif
> 
> which would normally be funny, except that it's not quite a joke.  :P

Nice visualisation! Was that yours?

> So you want to minimize hard disk usage as much as possible.  "Thrashing"
> is precisely the situation you do not want to have when running a large
> analysis.

Yes, thrashing is a disaster for performance. But I think that hard 
drive latency fails to demonstrate just how big a disaster. True, hard 
drive access time is about 100 times slower than RAM. But that's just a 
constant scale factor. What really, really kills performance with 
thrashing is that the memory manager is trying to move chunks of memory 
around, and the overall "move smaller blocks of memory around to make 
space for a really big block" algorithm ends up with quadratic (or 
worse!) performance.


-- 
Steven