[Numpy-discussion] Loading a > GB file into array

[Martin Spacek]

Sebastian Haase wrote:
> b) To my knowledge, any OS Linux, Windows an OSX can max. allocate
> about 1GB of data - assuming you have a 32 bit machine.
> The actual numbers I measured varied  from about 700MB to maybe 1.3GB.
> In other words, you would be right at the limit.
> (For 64bit, you would have to make sure ALL parts are 64bit, e.g. The
> python version must be >=2.5, python must have been compiled using a
> 64-bit compiler *and* the windows version (XP-64))
> This holds true the same for physical ram allocation and for memmap
> allocation.
> My solution to this was to "wait" for the 64bit .... not tested yet ;-)

By the way, I installed 64-bit linux (ubuntu 7.10) on the same machine, 
and now numpy.memmap works like a charm. Slicing around a 15 GB file is fun!

--
Martin