[SciPy-user] partially reading a file...

[Travis E. Oliphant]

fred wrote:
> Travis E. Oliphant a écrit :
>   
>> fred wrote:
>>     
>>> Hi,
>>>
>>> Let's say I want to read a (binary) file which contains a nx*ny*nz array.
>>>
>>> Is it possible to read a "sub-array" from this file, ie each block of 
>>> (nx/4, ny/4, nz/4) for instance, without loading the whole file ?
>>>   
>>>       
>> An easy way to do this which forces the operating system to do the work 
>> of partial loading is to use a memory mapped file as the source of the 
>> array (i.e. a memmap array). 
>>
>> Then, selecting out a block is as simple as slicing.
>>     
> Maybe I had to mention this: the aim is to cut in several files a 
> "large" data file, _bigger_ than total available memory amount.
>   
Absolutely memory mapping still applies --- it's a perfect application 
for it.  But, you will probably need a 64-bit system.

Memory mapping is how the OS handles "virtual memory" which uses disk 
space to increase main memory.  You are just using that idea directly 
with a memory mapped file.

-Travis