[Numpy-discussion] Re : Saving an array on disk to free memory - Pickling

[Francesc Alted]

A Tuesday 18 May 2010 08:57:47 Jean-Baptiste Rudant escrigué:
> Thank you very much for the help.
> 
> But I was more looking for some coding solution (furthermore, I'm not using
>  Linux). My point in not to make some real arrays looking like they are
>  saved on files (and use for it some files in memory), but at the contrary,
>  to make some "fake" arrays, saved on disk, to look like real arrays. In
>  other words, to make them behave like if they were inherited from
>  ndarrays.
> 
> In pytables, if you use my_node[:] it returns an array. It's part of what I
>  want to do. Plus :
>  - if I call my_node[:] ten times, only the first call will read to disk
>  (it matters, even if pytables is very very fast) ;

Well, the second time that you read from disk, pytables will also will read 
from memory instead.  Indeed, it is the OS filesystem cache who will do the 
trick, and the speed may be not exactly the same as a pure numpy array, but 
hey, it should be very close and with zero implementation cost.

Moreover, if you have to do arithmetic computations with your arrays, you can 
make use of `tables.Expr()` module that can perform them generally faster than 
using pure numpy (for example, see http://pytables.org/moin/ComputingKernel).

-- 
Francesc Alted