[Numpy-discussion] create a numpy array of images
Chris.Barker at noaa.gov
Wed Feb 2 12:12:47 EST 2011
> It seems that using 64 bit python is the solution.
It's certainly the easy way to access a lot of memory -- and memory is
cheap these days.
> But the thing is i would
> compile my code and wanna distribute it to the clients..
I don't think 64 bit gets in the way of that -- except that it will only
runon 64 bit systems, which may be an issue.
> only reason why i want to work on 32 bit system. Sturla, how I can make it
> sure that some part of the data is kept on the disk and only the necessary
> one in the memory; as this seems to be a solution to my problem.
You can "roll your own" and have a disk cache of some sort -- it would
be pretty easy to store each image in a *.npz file and load the up as
you need them.
But it would probably be even easier to use one of the hdf-based
libarries, such as pytables -- I htink it will do it all for you.
One other option, that I've never tried, is carray, which is an array
compressed in memory. Depending on your images, perhaps they would
compress a lot (or not ....):
> As i said i
> want a 3d visualization out of the numpy array. it works fine for the
> downsampled dataset. And to visualize, i have to convert the 16bit data into
> 8bit as PIL doesnt support 16 bit data.
It's unclear to me what you native data really is: 16 bitgreyscale? 8bit
greyscale? either one should fit OK into 32 bit memory, and if 8bit is
accurate enough foryour needs, then it should be pretty easy.
> stack = numpy.empty((120, 1024, 1024))
numpy defaults to double precision float, np.float64, i.e. 8 bytes per
element -- you probably don't want that if you are concerned about
memoery use, and have 8 or 16 bit greyscale images. Try:
stack = np.empty((120, 1024, 1024), dtype=np.uint8) # (or dtype=np.uint16)
> i = 0
> for f in os.listdir(dirr):
> im = Image.open(f)
> im = im.convert("L")
you ight want to try mode "I". That should give you a 32 bit integer
grey scale, which should hold all the 16 bit data without loss -- then
you can convert to 16 bit when you bring it into numpy.
> a = numpy.asarray(im)
> print a.dtype
what does this print? it should be np.uint8
> stack[i] = a
here, if a is a uint8, numpy will convert it to a float64, to fit into
array a -- that's why you want to set the dtype of array a when you
> one more thing, it really doesnt work for tiff files at all, i have to
> convert them into jpgs as a prior step to this.
probably not the best choice either, jpeg is lossy -- is will smear
things out a bit, which you may not want. It also only holds 24 bit RGB
(I think), which is both a waste and will lose information from 16 bit
greyscale. If you have to convert, try PNG, though I'm not sure if it
handles 16 bit greyscale, either.
I'd look at a lib that can read tiff properly -- some have been
suggested here, and you can also use GDAL, which is meant for
geo-referenced data, but you can ignore the geo information and just get
an image if you want.
Christopher Barker, Ph.D.
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker at noaa.gov
More information about the NumPy-Discussion