<font color="#3366ff"><font face="verdana,sans-serif">Hi all,</font></font><div><font color="#3366ff"><font face="verdana,sans-serif"><br></font></font></div><div><font color="#3366ff"><font face="verdana,sans-serif">It seems that using 64 bit python is the solution. But the thing is i would compile my code and wanna distribute it to the clients.. and that is the only reason why i want to work on 32 bit system. Sturla, how I can make it sure that some part of the data is kept on the disk and only the necessary one in the memory; as this seems to be a solution to my problem. As i said i want a 3d visualization out of the numpy array. it works fine for the downsampled dataset. And to visualize, i have to convert the 16bit data into 8bit as PIL doesnt support 16 bit data.. the only thing i do to create my array is this:</font></font></div>
<div><font color="#3366ff"><font face="verdana,sans-serif"><br></font></font></div><div><font face="verdana,sans-serif" color="#666666"><div>stack = numpy.empty((120, 1024, 1024))</div><div><br></div><div>i = 0</div><div>
<br></div><div>os.chdir(dirr)</div><div>for f in os.listdir(dirr):</div><div><br></div><div> im = Image.open(f)</div><div> im = im.convert("L")</div><div> a = numpy.asarray(im)</div><div> print a.dtype</div>
<div> stack[i] = a</div><div> i += 1</div></font></div><div><font color="#3366ff"><font face="verdana,sans-serif"><br></font></font></div><div><font color="#3366ff"><font face="verdana,sans-serif">one more thing, it really doesnt work for tiff files at all, i have to convert them into jpgs as a prior step to this. and it at max lets me create an array for around 60 slices only, where as my requirement would be around 100 to 200 images..</font></font></div>
<div><font class="Apple-style-span" color="#3366FF" face="verdana, sans-serif">any ideas? can diagnose the problem??</font></div><div><font class="Apple-style-span" color="#3366FF" face="verdana, sans-serif"><br></font></div>
<div><font class="Apple-style-span" color="#3366FF" face="verdana, sans-serif">thanks a lot.. asmi</font></div><div><br><div class="gmail_quote">On Tue, Feb 1, 2011 at 8:57 PM, Francesc Alted <span dir="ltr"><<a href="mailto:faltet@pytables.org">faltet@pytables.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">A Tuesday 01 February 2011 19:58:16 Sturla Molden escrigué:<br>
<div class="im">> Den 01.02.2011 18:58, skrev Christopher Barker:<br>
> > But if you really have big collections of images, you might try<br>
> > memory mapped arrays -- as Sturla pointed out they wont' let you<br>
> > create monster arrays on a 32 bit python,<br>
><br>
> But they will on 64 bit Python :D We can just memory map a temporary<br>
> file, and fake as much "memory" as we need. The OS will optimize the<br>
> disk access.<br>
<br>
</div>What do you mean by 'optimize the disk access'? One of the drawbacks of<br>
memory mapped files is precisely that OS cannot distinguish between data<br>
that belongs to 'disk' and data that belongs to 'memory'. This normally<br>
introduces extreme slowness in other programs when datasets in files<br>
exceeds physical memory but have to loaded by OS -- the reason being<br>
that OS swaps out most of programs/shared libraries that were in memory<br>
in order to be able to load new 'disk' data.<br>
<br>
The other important drawback of the memory mapped files is that you need<br>
to have at very least an amount of virtual memory that is enough to keep<br>
all of these data files. In general, you only have virtual memory that<br>
is between 1.5x and 2x the physical memory (having more than this is<br>
generally regarded as a waste of disk space).<br>
<br>
This is why I prefer very much reading directly from a file: the OS in<br>
this case is able to distinguish between data belonging to 'disk' and<br>
data belonging to 'memory'. Is in this case when the OS can really<br>
optimize disk access (unless you have complicated setups).<br>
<div class="im"><br>
> Also consider that hardware is cheap compared to labour,<br>
> at least in Europe. Is programming for memory limited 32 bit Python<br>
> worth the effort?<br>
<br>
</div>--<br>
<font color="#888888">Francesc Alted<br>
</font><div><div></div><div class="h5">_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
</div></div></blockquote></div><br><br clear="all"><br>
</div>