[Numpy-discussion] Possible roadmap addendum: building better text file readers

[Lluís]

Erin Sheldon writes:
[...]
> This was why I essentially wrote my own memmap like interface with
> recfile, the code I'm converting.  It allows working with columns and
> rows without loading large chunks of memory.
[...]

This sounds like at any point in time you only have one part of the array mapped
into the application.

My question is then, why would you manually implement the buffering? The OS
should already take care of that by unmapping pages when it's short on physical
memory, and faulting pages in when you access them.

This reminds me of some previous discussion about making the ndarray API more
friendly to code that wants to manage the underlying storage, from mmap'ing it
to handling compressed storage. Are there any news on that front?


Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth