thanks again for you response. I must still be doing something wrong!!<br>both options resulted in :<br>the TSFC_avg is: [-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --<br> -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --<br>
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --<br> -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --<br> -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --<br>
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- <br><br>1st option:<br><br>slice_counter=0<br><br>for filename in netCDF_list:<br> ncfile=netCDF4.Dataset(filename)<br> TSFC=ncfile.variables['T_SFC'][:]<br>
fillvalue=ncfile.variables['T_SFC']._FillValue<br> TSFC=MA.masked_values(TSFC, fillvalue)<br> TSFCWithOutNan=[]<br> for a in TSFC:<br> indexnonNaN=N.isfinite(a)<br> SliceofTotoWithoutNan=a[indexnonNaN]<br>
print SliceofTotoWithoutNan<br> TSFCWithOutNan.append(SliceofTotoWithoutNan)<br> for i in xrange(0,len(TSFCWithOutNan)-1,1):<br> slice_counter +=1<br> try:<br>
running_sum=N.add(running_sum, TSFCWithOutNan[i])<br> except NameError:<br> print "Initiating the running total of my variable..."<br> running_sum=N.array(TSFCWithOutNan[i])<br>
<br>TSFC_avg=N.true_divide(running_sum, slice_counter)<br>N.set_printoptions(threshold='nan')<br>print "the TSFC_avg is:", TSFC_avg<br><br><br><br>the 2nd option :<br><br>for filename in netCDF_list:<br>
ncfile=netCDF4.Dataset(filename)<br> TSFC=ncfile.variables['T_SFC'][:]<br> fillvalue=ncfile.variables['T_SFC']._FillValue<br> TSFC=MA.masked_values(TSFC, fillvalue)<br><br> slice_counter=0<br>
for a in TSFC:<br> indexnonNaN=N.isfinite(a)<br> SliceofTotoWithoutNan=a[indexnonNaN]<br> slice_counter +=1<br> try:<br> running_sum=N.add(running_sum, SliceofTotoWithoutNan)<br>
except NameError:<br> print "Initiating the running total of my variable..."<br> running_sum=N.array(SliceofTotoWithoutNan)<br><br>TSFC_avg=N.true_divide(running_sum, slice_counter)<br>
N.set_printoptions(threshold='nan')<br>print "the TSFC_avg is:", TSFC_avg<br><br><br><br><br><br><div class="gmail_quote">On Tue, Dec 6, 2011 at 2:31 PM, Xavier Barthelemy <span dir="ltr"><<a href="mailto:xabart@gmail.com">xabart@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div>Well, I would see solutions:<br></div><div>1- to keep how your code is, withj a python list (you can stack numpy arrays if they have the same dimensions):</div>
<div><br></div><div><div class="im">
<div>for filename in netCDF_list:<br> ncfile=netCDF4.Dataset(filename)<br>
TSFC=ncfile.variables['T_SFC'][:]<br> fillvalue=ncfile.variables['T_SFC']._FillValue<br> TSFC=MA.masked_values(TSFC, fillvalue)<br></div></div><div> TSFCWithOutNan=[]</div><div class="im">
for a in TSFC:<br> indexnonNaN=N.isfinite(a)<br>
SliceofTotoWithoutNan=a[indexnonNaN]<br> print SliceofTotoWithoutNan</div></div><div> TSFCWithOutNan
.append(
SliceofTotoWithoutNan
)<br> <div><br><br> for i in xrange(0,len(TSFCWithOutNan
)-1,1):<div class="im"><br> slice_counter +=1<br>
#print slice_counter<br> try:<br></div> running_sum=N.add(running_sum, TSFCWithOutNan
[i])<div class="im"><br> except NameError:<br> print "Initiating the running total of my variable..."<br></div>
running_sum=N.array(TSFCWithOutNan
[i])<br></div></div><div>...<br></div><div><br></div><div>or 2- everything in the same loop:</div><div> </div><div>slice_counter
=0</div><div class="gmail_quote"><div class="im">
for a in TSFC:<br> indexnonNaN=N.isfinite(a)<br>
SliceofTotoWithoutNan=a[indexnonNaN]<br></div><div class="im"> slice_counter +=1<br>
#print slice_counter<br> try:<br></div> running_sum=N.add(running_sum,
SliceofTotoWithoutNan
)<div class="im"><br> except NameError:<br> print "Initiating the running total of my variable..."<br></div>
running_sum=N.array(
SliceofTotoWithoutNan
)
<br><div class="im">
TSFC_avg=N.true_divide(running_sum, slice_counter)<br>N.set_printoptions(threshold='nan')<br>print "the TSFC_avg is:", TSFC_avg
<br></div></div><div class="gmail_quote"><br></div><div class="gmail_quote">See if it works. it is just a rapid guess</div><div class="gmail_quote"><span class="HOEnZb"><font color="#888888">Xavier</font></span><div><div class="h5">
<br><br>for dir in glob.glob(MainFolder + '*/01/')+ glob.glob(MainFolder + '*/02/')+ glob.glob(MainFolder + '*/12/'):<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
#print dir<div><br> for ncfile in glob.glob(dir + '*.nc'):<br> netCDF_list.append(ncfile)<br><br>slice_counter=0<br>print netCDF_list<br>for filename in netCDF_list:<br> ncfile=netCDF4.Dataset(filename)<br>
TSFC=ncfile.variables['T_SFC'][:]<br> fillvalue=ncfile.variables['T_SFC']._FillValue<br> TSFC=MA.masked_values(TSFC, fillvalue)<br></div> for a in TSFC:<br> indexnonNaN=N.isfinite(a)<br>
SliceofTotoWithoutNan=a[indexnonNaN]<br> print SliceofTotoWithoutNan<br> TSFC=SliceofTotoWithoutNan<div><br><br> for i in xrange(0,len(TSFC)-1,1):<br> slice_counter +=1<br>
#print slice_counter<br> try:<br> running_sum=N.add(running_sum, TSFC[i])<br> except NameError:<br> print "Initiating the running total of my variable..."<br>
running_sum=N.array(TSFC[i])<br><br>TSFC_avg=N.true_divide(running_sum, slice_counter)<br>N.set_printoptions(threshold='nan')<br>print "the TSFC_avg is:", TSFC_avg<br><br>
<br><br><br></div><div><div><div class="gmail_quote">On Tue, Dec 6, 2011 at 9:50 AM, Xavier Barthelemy <span dir="ltr"><<a href="mailto:xabart@gmail.com" target="_blank">xabart@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div>Hi, <br></div><div>I don't know if it is the best choice, but this is what I do in my code:</div><div><br></div><div>for each slice:</div><div> indexnonNaN=np.isfinite(SliceOf Toto)</div><div> SliceOf TotoWithoutNan=
SliceOf Toto
[indexnonNaN]<br></div><div><br></div><div>and then perform all operation I want o on the last array.</div><div><br></div><div>i hope it does answer your question</div><div><br></div><div>Xavier</div><div><br></div><br><div class="gmail_quote">
2011/12/6 questions anon <span dir="ltr"><<a href="mailto:questions.anon@gmail.com" target="_blank">questions.anon@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div><div>
Maybe I am asking the wrong question or could go about this another way.<br>I have thousands of numpy arrays to flick through, could I just identify which arrays have NAN's and for now ignore the entire array. is there a simple way to do this?<br>
any feedback will be greatly appreciated. <br><div><div><br><div class="gmail_quote">On Thu, Dec 1, 2011 at 12:16 PM, questions anon <span dir="ltr"><<a href="mailto:questions.anon@gmail.com" target="_blank">questions.anon@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">I am trying to calculate the mean across many netcdf files. I cannot use numpy.mean because there are too many files to concatenate and I end up with a memory error. I have enabled the below code to do what I need but I have a few nan values in some of my arrays. Is there a way to ignore these somewhere in my code. I seem to face this problem often so I would love a command that ignores blanks in my array before I continue on to the next processing step.<br>
Any feedback is greatly appreciated.<br><br><br>netCDF_list=[]<br>for dir in glob.glob(MainFolder + '*/01/')+ glob.glob(MainFolder + '*/02/')+ glob.glob(MainFolder + '*/12/'):<br> for ncfile in glob.glob(dir + '*.nc'):<br>
netCDF_list.append(ncfile)<br><br>slice_counter=0<br>print netCDF_list<br><br>for filename in netCDF_list:<br> ncfile=netCDF4.Dataset(filename)<br> TSFC=ncfile.variables['T_SFC'][:]<br> fillvalue=ncfile.variables['T_SFC']._FillValue<br>
TSFC=MA.masked_values(TSFC, fillvalue)<br> for i in xrange(0,len(TSFC)-1,1):<br> slice_counter +=1<br> #print slice_counter<br> try:<br> running_sum=N.add(running_sum, TSFC[i])<br>
except NameError:<br> print "Initiating the running total of my variable..."<br> running_sum=N.array(TSFC[i])<br><br>TSFC_avg=N.true_divide(running_sum, slice_counter)<br>
N.set_printoptions(threshold='nan')<br>print "the TSFC_avg is:", TSFC_avg<br><br>
</blockquote></div><br>
</div></div><br></div></div><div>_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
<br></div></blockquote></div><span><font color="#888888"><br><br clear="all"><br>-- <br> « Quand le gouvernement viole les droits du peuple, l'insurrection est, pour le peuple et pour chaque portion du peuple, le plus sacré des droits et le plus indispensable des devoirs »<br>
<br>Déclaration des droits de l'homme et du citoyen, article 35, 1793<br>
</font></span><br>_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
<br></blockquote></div><br>
</div></div><br>_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
<br></blockquote></div></div></div><div class="HOEnZb"><div class="h5"><br><br clear="all"><br>-- <br> « Quand le gouvernement viole les droits du peuple, l'insurrection est, pour le peuple et pour chaque portion du peuple, le plus sacré des droits et le plus indispensable des devoirs »<br>
<br>Déclaration des droits de l'homme et du citoyen, article 35, 1793<br>
</div></div><br>_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
<br></blockquote></div><br>