<div>Well, I would see solutions:<br></div><div>1- to keep how your code is, withj a python list (you can stack numpy arrays if they have the same dimensions):</div><div><br></div><div>
<div class="im">for filename in netCDF_list:<br> ncfile=netCDF4.Dataset(filename)<br>
TSFC=ncfile.variables['T_SFC'][:]<br> fillvalue=ncfile.variables['T_SFC']._FillValue<br> TSFC=MA.masked_values(TSFC, fillvalue)<br></div><div class="im"> TSFCWithOutNan=[]</div>
for a in TSFC:<br> indexnonNaN=N.isfinite(a)<br>
SliceofTotoWithoutNan=a[indexnonNaN]<br> print SliceofTotoWithoutNan</div><div> TSFCWithOutNan
.append(
SliceofTotoWithoutNan
)<br> <div class="im HOEnZb"><br><br> for i in xrange(0,len(TSFCWithOutNan
)-1,1):<br> slice_counter +=1<br>
#print slice_counter<br> try:<br> running_sum=N.add(running_sum, TSFCWithOutNan
[i])<br> except NameError:<br> print "Initiating the running total of my variable..."<br>
running_sum=N.array(TSFCWithOutNan
[i])<br></div></div><div>...<br></div><div><br></div><div>or 2- everything in the same loop:</div><div> </div><div>slice_counter
=0</div><div class="gmail_quote">
for a in TSFC:<br> indexnonNaN=N.isfinite(a)<br>
SliceofTotoWithoutNan=a[indexnonNaN]<br> slice_counter +=1<br>
#print slice_counter<br> try:<br> running_sum=N.add(running_sum,
SliceofTotoWithoutNan
)<br> except NameError:<br> print "Initiating the running total of my variable..."<br>
running_sum=N.array(
SliceofTotoWithoutNan
)
<br>
TSFC_avg=N.true_divide(running_sum, slice_counter)<br>N.set_printoptions(threshold='nan')<br>print "the TSFC_avg is:", TSFC_avg
<br></div><div class="gmail_quote"><br></div><div class="gmail_quote">See if it works. it is just a rapid guess</div><div class="gmail_quote">Xavier<br><br>for dir in glob.glob(MainFolder + '*/01/')+ glob.glob(MainFolder + '*/02/')+ glob.glob(MainFolder + '*/12/'):<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
#print dir<div class="im"><br> for ncfile in glob.glob(dir + '*.nc'):<br> netCDF_list.append(ncfile)<br><br>slice_counter=0<br>print netCDF_list<br>for filename in netCDF_list:<br> ncfile=netCDF4.Dataset(filename)<br>
TSFC=ncfile.variables['T_SFC'][:]<br> fillvalue=ncfile.variables['T_SFC']._FillValue<br> TSFC=MA.masked_values(TSFC, fillvalue)<br></div> for a in TSFC:<br> indexnonNaN=N.isfinite(a)<br>
SliceofTotoWithoutNan=a[indexnonNaN]<br> print SliceofTotoWithoutNan<br> TSFC=SliceofTotoWithoutNan<div class="im HOEnZb"><br><br> for i in xrange(0,len(TSFC)-1,1):<br> slice_counter +=1<br>
#print slice_counter<br> try:<br> running_sum=N.add(running_sum, TSFC[i])<br> except NameError:<br> print "Initiating the running total of my variable..."<br>
running_sum=N.array(TSFC[i])<br><br>TSFC_avg=N.true_divide(running_sum, slice_counter)<br>N.set_printoptions(threshold='nan')<br>print "the TSFC_avg is:", TSFC_avg<br><br>
<br><br><br></div><div class="HOEnZb"><div class="h5"><div class="gmail_quote">On Tue, Dec 6, 2011 at 9:50 AM, Xavier Barthelemy <span dir="ltr"><<a href="mailto:xabart@gmail.com" target="_blank">xabart@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>Hi, <br></div><div>I don't know if it is the best choice, but this is what I do in my code:</div><div><br></div><div>for each slice:</div><div> indexnonNaN=np.isfinite(SliceOf Toto)</div><div> SliceOf TotoWithoutNan=
SliceOf Toto
[indexnonNaN]<br></div><div><br></div><div>and then perform all operation I want o on the last array.</div><div><br></div><div>i hope it does answer your question</div><div><br></div><div>Xavier</div><div><br></div><br><div class="gmail_quote">
2011/12/6 questions anon <span dir="ltr"><<a href="mailto:questions.anon@gmail.com" target="_blank">questions.anon@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div><div>
Maybe I am asking the wrong question or could go about this another way.<br>I have thousands of numpy arrays to flick through, could I just identify which arrays have NAN's and for now ignore the entire array. is there a simple way to do this?<br>
any feedback will be greatly appreciated. <br><div><div><br><div class="gmail_quote">On Thu, Dec 1, 2011 at 12:16 PM, questions anon <span dir="ltr"><<a href="mailto:questions.anon@gmail.com" target="_blank">questions.anon@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I am trying to calculate the mean across many netcdf files. I cannot use numpy.mean because there are too many files to concatenate and I end up with a memory error. I have enabled the below code to do what I need but I have a few nan values in some of my arrays. Is there a way to ignore these somewhere in my code. I seem to face this problem often so I would love a command that ignores blanks in my array before I continue on to the next processing step.<br>
Any feedback is greatly appreciated.<br><br><br>netCDF_list=[]<br>for dir in glob.glob(MainFolder + '*/01/')+ glob.glob(MainFolder + '*/02/')+ glob.glob(MainFolder + '*/12/'):<br> for ncfile in glob.glob(dir + '*.nc'):<br>
netCDF_list.append(ncfile)<br><br>slice_counter=0<br>print netCDF_list<br><br>for filename in netCDF_list:<br> ncfile=netCDF4.Dataset(filename)<br> TSFC=ncfile.variables['T_SFC'][:]<br> fillvalue=ncfile.variables['T_SFC']._FillValue<br>
TSFC=MA.masked_values(TSFC, fillvalue)<br> for i in xrange(0,len(TSFC)-1,1):<br> slice_counter +=1<br> #print slice_counter<br> try:<br> running_sum=N.add(running_sum, TSFC[i])<br>
except NameError:<br> print "Initiating the running total of my variable..."<br> running_sum=N.array(TSFC[i])<br><br>TSFC_avg=N.true_divide(running_sum, slice_counter)<br>
N.set_printoptions(threshold='nan')<br>print "the TSFC_avg is:", TSFC_avg<br><br>
</blockquote></div><br>
</div></div><br></div></div><div>_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
<br></div></blockquote></div><span><font color="#888888"><br><br clear="all"><br>-- <br> « Quand le gouvernement viole les droits du peuple, l'insurrection est, pour le peuple et pour chaque portion du peuple, le plus sacré des droits et le plus indispensable des devoirs »<br>
<br>Déclaration des droits de l'homme et du citoyen, article 35, 1793<br>
</font></span><br>_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
<br></blockquote></div><br>
</div></div><br>_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br> « Quand le gouvernement viole les droits du peuple, l'insurrection est, pour le peuple et pour chaque portion du peuple, le plus sacré des droits et le plus indispensable des devoirs »<br>
<br>Déclaration des droits de l'homme et du citoyen, article 35, 1793<br>