Thanx for the fast response Robert ;)<br><br>I changed my code to use the slice:<br><div style="margin-left: 40px;">
E = data[6::9]</div>It is indeed faster and less eat less memory. Great.<br><br>Thanx for the endiannes! I knew there was something like this ;) I suspect that, in '>f8', "f" means float and "8" means 8 bytes?<br>
<br>From some benchmarks, I see that the slowest thing is disk access. It can slow the displaying of data from around 1sec (when data is in os cache or buffer) to 8sec.<br><br>So the next step would be to only read the needed data from the binary file... Is it possible to read from a file with a slice? So instead of:<br>
<div style="margin-left: 40px;">data = numpy.fromfile(file=f, dtype=float_dtype, count=9*Stot)<br>E = data[6::9]<br></div>maybe something like:<br><div style="margin-left: 40px;">E = numpy.fromfile(file=f, dtype=float_dtype, count=9*Stot, slice=6::9)<br>
</div><br>Thank you!<br><br><br><div><span class="gmail_quote">2008/4/3, Robert Kern <<a href="mailto:robert.kern@gmail.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">robert.kern@gmail.com</a>>:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
On Thu, Apr 3, 2008 at 3:30 PM, Nicolas Bigaouette<br> <<a href="mailto:nbigaouette@gmail.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">nbigaouette@gmail.com</a>> wrote:<br> > Hi,<br>
><br> > I have a C program which outputs large (~GB) files. It is a simple binary<br>
> dump of an array of structure containing 9 doubles. You can see this as a<br> > double 1D array of size 9*Stot (Stot being the allocated size of the array<br> > of structure). The 1D array represents a 3D array (Sx * Sy * Sz = Stot)<br>
> containing 9 values per cell.<br> ><br> > I want to read these files in the most efficient way possible, and I would<br> > like to have your insight on this.<br> ><br> > Right now, the fastest way I found was:<br>
> imzeros = zeros((Sy,Sz),dtype=float64,order='C')<br> > imex = imshow(imzeros)<br> > f = open(filename, 'rb')<br> > data = numpy.fromfile(file=f, dtype=numpy.float64, count=9*Stot)<br> > mask_Ex = numpy.arange(6,9*Stot,9)<br>
<br> <br>This is something you can do much, much more efficiently by using a<br> slice instead of indexing with an integer array.<br> <br><br> > Ex = data[mask].reshape((Sz,Sy,Sx), order='C').transpose()<br> > imex.set_array(squeeze(Ex3D[:,:,z]))<br>
><br> > The arrays will be big, so everything should be well optimized. I have<br> > multiple questions:<br> ><br> > 1) Should I change this:<br> > Ex = data[mask].reshape((Sz,Sy,Sx), order='C').transpose()<br>
> imex.set_array(squeeze(Ex3D[:,:,z]))<br> > to:<br> > imex.set_array(squeeze(data[mask].reshape((Sz,Sy,Sx),<br> > order='C').transpose()[:,:,z]))<br> > I mean, is I don't use a temporary variable, will it be faster or less<br>
> memory hungry?<br> <br> <br>No. The temporary exists whether you give it a name or not. If you use<br> data[6::9] instead of data[mask], you won't be using any extra memory<br> at all. The arrays will just be views into the original array.<br>
<br><br> > 2) If not, is the operation "Ex = " update the variable data or create<br> > another one?<br> <br> <br>It just reassigns the name "Ex" to a different object specified on the<br> right-hand side of the assignment. The relevant question is whether<br>
expression on the right-hand side takes up more memory.<br> <br><br> > Ideally I would like to only update it. Maybe this would be<br> > better:<br> ><br> > Ex[:,:,:] = data[mask].reshape((Sz,Sy,Sx), order='C').transpose()Would it?<br>
<br> <br>If you use data[6::9] instead of data[mask], you should just use "Ex =<br> " since no new memory will be used on the RHS.<br> <br><br> > 3) The machine where the code will be run might be big-endian. Is there a<br>
> way for python to read the big-endian file and "translate" it automatically<br> > to little-endian? Something like "numpy.fromfile(file=f,<br> > dtype=numpy.float64, count=9*Stot, endianness='big')"?<br>
<br> <br>dtype=numpy.dtype('>f8')<br> <br> --<br> Robert Kern<br> <br> "I have come to believe that the whole world is an enigma, a harmless<br> enigma that is made terrible by our own mad attempt to interpret it as<br>
though it had an underlying truth."<br> -- Umberto Eco<br> _______________________________________________<br> Numpy-discussion mailing list<br> <a href="mailto:Numpy-discussion@scipy.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">Numpy-discussion@scipy.org</a><br>
<a href="http://projects.scipy.org/mailman/listinfo/numpy-discussion" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://projects.scipy.org/mailman/listinfo/numpy-discussion</a><br> </blockquote>
</div><br>