Just a few extra tests on my side pushing the limits of my system memory:<div><br></div><div>In [34]: k = np.zeros((21601, 10801, 3), dtype='int16')</div><div>k ndarray 21601x10801x3: 699937203 elems, type `int16`, 1399874406 bytes (1335 Mb)</div>
<div><br></div><div>And for the first time my memory explodes with a hard kernel crash:</div><div><br></div><div><div>In [36]: k = np.zeros((21601, 10801, 13), dtype='int16')</div><div><br></div><div>Message from syslogd@ccn at Aug 3 10:51:43 ...</div>
<div> kernel:[48715.531155] ------------[ cut here ]------------</div><div><br></div><div>Message from syslogd@ccn at Aug 3 10:51:43 ...</div><div> kernel:[48715.531163] invalid opcode: 0000 [#1] SMP </div><div><br></div>
<div>Message from syslogd@ccn at Aug 3 10:51:43 ...</div><div> kernel:[48715.531166] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map</div><div><br></div><div>Message from syslogd@ccn at Aug 3 10:51:43 ...</div>
<div> kernel:[48715.531253] Stack:</div><div><br></div><div>Message from syslogd@ccn at Aug 3 10:51:43 ...</div><div> kernel:[48715.531265] Call Trace:</div><div><br></div><div>Message from syslogd@ccn at Aug 3 10:51:43 ...</div>
<div> kernel:[48715.531332] Code: be 33 01 00 00 48 89 fb 48 c7 c7 67 31 7a 81 e8 b0 2d f1 ff e8 90 f2 33 00 48 89 df e8 86 db 00 00 48 83 bb 60 01 00 00 00 74 02 <0f> 0b 48 8b 83 10 02 00 00 a8 20 75 02 0f 0b a8 40 74 02 0f 0b </div>
<div><br></div><br><div class="gmail_quote">On Wed, Aug 3, 2011 at 10:46 AM, Gökhan Sever <span dir="ltr"><<a href="mailto:gokhansever@gmail.com">gokhansever@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Here are my values for your comparison:<div><br></div><div><a href="http://test.nc" target="_blank">test.nc</a> file is about 715 MB. The details are below:<br><div><br></div><div><div>In [21]: netCDF4.__version__</div><div>
Out[21]: '0.9.4'</div>
<div><br></div><div>In [22]: np.__version__</div><div>Out[22]: '2.0.0.dev-b233716'</div><div><br></div><div>In [23]: from netCDF4 import Dataset</div><div><br></div><div>In [24]: f = Dataset("<a href="http://test.nc" target="_blank">test.nc</a>")</div>
<div><br></div><div>In [25]: f.variables['reflectivity'].shape</div><div>Out[25]: (6, 18909, 506)</div><div><br></div><div>In [26]: f.variables['reflectivity'].size</div><div>Out[26]: 57407724</div><div><br>
</div><div>In [27]: f.variables['reflectivity'][:].dtype</div><div>Out[27]: dtype('float32')</div><div><br></div><div>In [28]: timeit z = f.variables['reflectivity'][:]</div><div>1 loops, best of 3: 731 ms per loop</div>
<div><br></div><div>How long it takes in your side to read that big array?</div><br><div class="gmail_quote"><div><div></div><div class="h5">On Wed, Aug 3, 2011 at 10:30 AM, Kiko <span dir="ltr"><<a href="mailto:kikocorreoso@gmail.com" target="_blank">kikocorreoso@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div></div><div class="h5">Hi.<br><br>I'm trying to read a big netcdf file (445 Mb) using netcdf4-python.<br>
<br>The data are described as:<br>
<i>The GEBCO gridded data set is stored in NetCDF as a one dimensional array of 2-byte signed integers that represent integer elevations in metres. <br>
The complete data set gives global coverage. It consists of 21601 x 10801 data values, one for each one minute of latitude and longitude for 233312401 points.<br>The data start at position 90°N, 180°W and are arranged in bands of 360 degrees x 60 points/degree + 1 = 21601 values. The data range eastward from 180°W longitude to 180°E longitude, i.e. the 180° value is repeated.</i><br>
<br>The problem is that it is very slow (or I am quite newbie).<br><br>Anyone has a suggestion to get these data in a numpy array in a faster way?<br><br>Thanks in advance.<br>
<br></div></div>_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br>Gökhan<br>
</div></div>
</blockquote></div><br><br clear="all"><br>-- <br>Gökhan<br>
</div>