I need some help understanding how to loop through many arrays to calculate the 95th percentile.<br>I can easily do this by using numpy.concatenate to make one big array and then finding the 95th percentile using numpy.percentile but this causes a memory error when I want to run this on 100's of netcdf files (see code below).<br>
Any alternative methods will be greatly appreciated.<br> <br><br>all_TSFC=[] <br>for (path, dirs, files) in os.walk(MainFolder):<br> for dir in dirs:<br> print dir<br> path=path+'/'<br> for ncfile in files:<br>
if ncfile[-3:]=='.nc':<br> print "dealing with ncfiles:", ncfile<br> ncfile=os.path.join(path,ncfile)<br> ncfile=Dataset(ncfile, 'r+', 'NETCDF4')<br>
TSFC=ncfile.variables['T_SFC'][:]<br> ncfile.close()<br> all_TSFC.append(TSFC)<br> <br>big_array=N.ma.concatenate(all_TSFC)<br>Percentile95th=N.percentile(big_array, 95, axis=0)<br>
<br><br>