[Numpy-discussion] create a numpy array of images
faltet at pytables.org
Tue Feb 1 14:57:57 EST 2011
A Tuesday 01 February 2011 19:58:16 Sturla Molden escrigué:
> Den 01.02.2011 18:58, skrev Christopher Barker:
> > But if you really have big collections of images, you might try
> > memory mapped arrays -- as Sturla pointed out they wont' let you
> > create monster arrays on a 32 bit python,
> But they will on 64 bit Python :D We can just memory map a temporary
> file, and fake as much "memory" as we need. The OS will optimize the
> disk access.
What do you mean by 'optimize the disk access'? One of the drawbacks of
memory mapped files is precisely that OS cannot distinguish between data
that belongs to 'disk' and data that belongs to 'memory'. This normally
introduces extreme slowness in other programs when datasets in files
exceeds physical memory but have to loaded by OS -- the reason being
that OS swaps out most of programs/shared libraries that were in memory
in order to be able to load new 'disk' data.
The other important drawback of the memory mapped files is that you need
to have at very least an amount of virtual memory that is enough to keep
all of these data files. In general, you only have virtual memory that
is between 1.5x and 2x the physical memory (having more than this is
generally regarded as a waste of disk space).
This is why I prefer very much reading directly from a file: the OS in
this case is able to distinguish between data belonging to 'disk' and
data belonging to 'memory'. Is in this case when the OS can really
optimize disk access (unless you have complicated setups).
> Also consider that hardware is cheap compared to labour,
> at least in Europe. Is programming for memory limited 32 bit Python
> worth the effort?
More information about the NumPy-Discussion