[Numpy-discussion] create a numpy array of images

Asmi Shah asmi.capri at gmail.com
Wed Feb 2 05:22:15 EST 2011

Hi all,

It seems that using 64 bit python is the solution. But the thing is i would
compile my code and wanna distribute it to the clients.. and that is the
only reason why i want to work on 32 bit system. Sturla, how I can make it
sure that some part of the data is kept on the disk and only the necessary
one in the memory; as this seems to be a solution to my problem. As i said i
want a 3d visualization out of the numpy array. it works fine for the
downsampled dataset. And to visualize, i have to convert the 16bit data into
8bit as PIL doesnt support 16 bit data.. the only thing i do to create my
array is this:

stack = numpy.empty((120, 1024, 1024))

i = 0

for f in os.listdir(dirr):

    im = Image.open(f)
    im = im.convert("L")
    a = numpy.asarray(im)
    print a.dtype
    stack[i] = a
    i += 1

one more thing, it really doesnt work for tiff files at all, i have to
convert them into jpgs as a prior step to this. and it at max lets me create
an array for around 60 slices only, where as my requirement would be around
100 to 200 images..
any ideas? can diagnose the problem??

thanks a lot.. asmi

On Tue, Feb 1, 2011 at 8:57 PM, Francesc Alted <faltet at pytables.org> wrote:

> A Tuesday 01 February 2011 19:58:16 Sturla Molden escrigué:
> > Den 01.02.2011 18:58, skrev Christopher Barker:
> > > But if you really have big collections of images, you might try
> > > memory mapped arrays -- as Sturla pointed out they wont' let you
> > > create monster arrays on a 32 bit python,
> >
> > But they will on 64 bit Python :D We can just memory map a temporary
> > file, and fake as much "memory" as we need. The OS will optimize the
> > disk access.
> What do you mean by 'optimize the disk access'?  One of the drawbacks of
> memory mapped files is precisely that OS cannot distinguish between data
> that belongs to 'disk' and data that belongs to 'memory'.  This normally
> introduces extreme slowness in other programs when datasets in files
> exceeds physical memory but have to loaded by OS -- the reason being
> that OS swaps out most of programs/shared libraries that were in memory
> in order to be able to load new 'disk' data.
> The other important drawback of the memory mapped files is that you need
> to have at very least an amount of virtual memory that is enough to keep
> all of these data files.  In general, you only have virtual memory that
> is between 1.5x and 2x the physical memory (having more than this is
> generally regarded as a waste of disk space).
> This is why I prefer very much reading directly from a file: the OS in
> this case is able to distinguish between data belonging to 'disk' and
> data belonging to 'memory'.  Is in this case when the OS can really
> optimize disk access (unless you have complicated setups).
> > Also consider that hardware is cheap compared to labour,
> > at least in Europe. Is programming for memory limited 32 bit Python
> > worth the effort?
> --
> Francesc Alted
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110202/657b3b5c/attachment.html>

More information about the NumPy-Discussion mailing list