<div dir="ltr"><div><div>Thanks Stephan,<br></div>It doesn't look like CDAT has 'daily' option - it has yearly, seasonal and monthly! I would need to look into IRIS more as it is new to me and I can't quiet figure out all the steps required for xray, although it looks great. <br>
<br></div>Another way around was after converting to localtime_day I could append the corresponding hourly arrays to a list, concatenate, calculate max and make the max equal to that localtime_day. Then I could delete everything in that list and repeat by looping though the hours of the next day and append to the empty list. Although I really don't know how to get this to work. <br>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, May 22, 2014 at 10:56 AM, Stephan Hoyer <span dir="ltr"><<a href="mailto:shoyer@gmail.com" target="_blank">shoyer@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hello anonymous,<div><br></div><div>I recently wrote a package "xray" (<a href="http://xray.readthedocs.org/" target="_blank">http://xray.readthedocs.org/</a>) specifically to make it easier to work with high-dimensional labeled data, as often found in NetCDF files. Xray has a groupby method for grouping over subsets of your data, which would seem well suited to what you're trying to do. Something like the following might work:</div>
<div><br></div><div><div>ds = xray.open_dataset(ncfile)</div><div>tmax = ds['temperature'].groupby('time.hour').max()</div></div><div><br></div><div>It also might be worth looking at other more data analysis packages, either more generic (e.g., pandas, <a href="http://pandas.pydata.org/" target="_blank">http://pandas.pydata.org/</a>) or weather/climate data specific (e.g., Iris, <a href="http://scitools.org.uk/iris/" target="_blank">http://scitools.org.uk/iris/</a> and CDAT, <a href="http://www2-pcmdi.llnl.gov/cdat/manuals/cdutil/cdat_utilities.html" target="_blank">http://www2-pcmdi.llnl.gov/cdat/manuals/cdutil/cdat_utilities.html</a>).</div>
<div><br></div><div>Cheers,</div><div>Stephan</div></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div class="h5">On Wed, May 21, 2014 at 5:27 PM, questions anon <span dir="ltr"><<a href="mailto:questions.anon@gmail.com" target="_blank">questions.anon@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><div dir="ltr"><br><div class="gmail_quote"><div dir="ltr"><div><div><div>
I have hourly 2D temperature data in a monthly netcdf and I would like to find the daily maximum temperature. The shape of the netcdf is (744, 106, 193)<br><br></div>I would like to use the year-month-day as a new list name (i.e. 2009-03-01, 2009-03-02....2009-03-31) and then add each of the hours worth of temperature data to each corresponding list. Therefore each new list should contain 24 hours worth of data and the shape should be (24,106,193) . This is the part I cannot seem to get to work. I am using datetime and then groupby to group by date but I am not sure how to use the output to make a new list name and then add the data for that day into that list. see below and attached for my latest attempt. Any feedback will be greatly appreciated. <br>
</div></div><div><div><div><div><p style="margin:0px;text-indent:0px"><br></p><p style="margin:0px;text-indent:0px"></p><p style="margin:0px;text-indent:0px"><br></p>
<p style="margin:0px;text-indent:0px">from netCDF4 import Dataset</p>
<p style="margin:0px;text-indent:0px">import numpy as np</p>
<p style="margin:0px;text-indent:0px">import matplotlib.pyplot as plt</p>
<p style="margin:0px;text-indent:0px">from mpl_toolkits.basemap import Basemap</p>
<p style="margin:0px;text-indent:0px">from netcdftime import utime</p>
<p style="margin:0px;text-indent:0px">from datetime import datetime as dt</p>
<p style="margin:0px;text-indent:0px">import os</p>
<p style="margin:0px;text-indent:0px">import gc</p>
<p style="margin:0px;text-indent:0px">from numpy import *</p>
<p style="margin:0px;text-indent:0px">import pytz</p>
<p style="margin:0px;text-indent:0px">from itertools import groupby</p>
<p style="margin:0px;text-indent:0px"><br></p>
<p style="margin:0px;text-indent:0px">MainFolder=r"/DATA/2009/03"</p><br><p style="margin:0px;text-indent:0px">dailydate=[]</p>
<p style="margin:0px;text-indent:0px">alltime=[]</p>
<p style="margin:0px;text-indent:0px">lists={}</p>
<p style="margin:0px;text-indent:0px"><br></p>
<p style="margin:0px;text-indent:0px"><br></p>
<p style="margin:0px;text-indent:0px">ncvariablename='T_SFC'</p>
<p style="margin:0px;text-indent:0px"><br></p>
<p style="margin:0px;text-indent:0px">for (path, dirs, files) in os.walk(MainFolder):</p>
<p style="margin:0px;text-indent:0px"> for ncfile in files:</p>
<p style="margin:0px;text-indent:0px"> print ncfile</p>
<p style="margin:0px;text-indent:0px"> fileext='.nc'</p>
<p style="margin:0px;text-indent:0px"> if ncfile.endswith(ncvariablename+'.nc'):</p>
<p style="margin:0px;text-indent:0px"> print "dealing with ncfiles:", path+ncfile </p>
<p style="margin:0px;text-indent:0px"> ncfile=os.path.join(path,ncfile)</p>
<p style="margin:0px;text-indent:0px"> ncfile=Dataset(ncfile, 'r+', 'NETCDF4')</p>
<p style="margin:0px;text-indent:0px"> variable=ncfile.variables[ncvariablename][:,:,:]</p>
<p style="margin:0px;text-indent:0px"> TIME=ncfile.variables['time'][:]</p>
<p style="margin:0px;text-indent:0px"> ncfile.close()</p>
<p style="margin:0px;text-indent:0px"> </p>
<p style="margin:0px;text-indent:0px"> for temp, time in zip((variable[:]),(TIME[:])): </p>
<p style="margin:0px;text-indent:0px"> cdftime=utime('seconds since 1970-01-01 00:00:00')</p>
<p style="margin:0px;text-indent:0px"> ncfiletime=cdftime.num2date(time)</p>
<p style="margin:0px;text-indent:0px"> timestr=str(ncfiletime)</p>
<p style="margin:0px;text-indent:0px"> utc_dt = dt.strptime(timestr, '%Y-%m-%d %H:%M:%S')</p>
<p style="margin:0px;text-indent:0px"> au_tz = pytz.timezone('Australia/Sydney')</p>
<p style="margin:0px;text-indent:0px"> local_dt = utc_dt.replace(tzinfo=pytz.utc).astimezone(au_tz)</p>
<p style="margin:0px;text-indent:0px"> alltime.append(local_dt)</p>
<p style="margin:0px;text-indent:0px"> </p>
<p style="margin:0px;text-indent:0px"> for k, g in groupby(alltime, key=lambda d: d.date()):</p>
<p style="margin:0px;text-indent:0px"> kstrp_local=k.strftime('%Y-%m-%d_%H')</p>
<p style="margin:0px;text-indent:0px"> klocal_date=k.strftime('%Y-%m-%d')</p>
<p style="margin:0px;text-indent:0px"> dailydate.append(klocal_date)</p>
<p style="margin:0px;text-indent:0px"> for n in dailydate:</p>
<p style="margin:0px;text-indent:0px"> lists[n]=[]</p>
<p style="margin:0px;text-indent:0px"> lists[n].append(temp) </p>
<p style="margin:0px;text-indent:0px"><br></p>
<p style="margin:0px;text-indent:0px"> </p>
<p style="margin:0px;text-indent:0px"> </p>
<p style="margin:0px;text-indent:0px">big_array=np.ma.concatenate(lists[n])</p>
<p style="margin:0px;text-indent:0px">DailyTemp=big_array.max(axis=0)</p></div></div></div></div></div>
</div><br></div>
<br></div></div>_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
<br></blockquote></div><br></div>
<br>_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
<br></blockquote></div><br></div>