excellent thank you for the warning, I will look into dictionaries alot more carefully now. I have some simple questions that I would like to ask straight away but will try and figure a few things out on my own first. <br>
<br>Thanks again!!<br><br><div class="gmail_quote">On Wed, May 9, 2012 at 11:16 AM, Andre' Walker-Loud <span dir="ltr"><<a href="mailto:walksloud@gmail.com" target="_blank">walksloud@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">dear anonymous questioner,<br>
<div class="im"><br>
> Excellent, thank you so much. I don't understand all the steps at this stage so I will need some time to go through it carefully but it works perfectly.<br>
> Thanks again!<br>
<br>
</div>words of caution - you will notice that the way I constructed the data file - I assumed what the input file would look like (they come chronologically and all data are present - no missing years or months). While this might be safe for a data file you constructed, and one that is short enough to read - this is generally a very bad habit - hence my encouraging you to figure out how to make dictionaries.<br>
<br>
Imagine how you would read a file that you got from a colleague, which was so long you can not look by eye and see that it is intact, or perhaps you know that in 1983, the month of June is missing as well as some other holes in the data. And perhaps your colleague decided, since those data are missing, just don't write the data to the file, so instead of having<br>
<br>
1983 May 2.780009889<br>
1983 June nan<br>
1983 July 0.138150181<br>
<br>
you have<br>
<br>
1983 May 2.780009889<br>
1983 July 0.138150181<br>
<br>
now the little loop I showed you will fail to place the data in the correct place in your numpy array, and you will get all your averaging and other analysis wrong.<br>
<br>
<br>
Instead - you can create dictionaries for the years and months. Then when you read in the data, you can grab this info to correctly place it in the right spot<br>
<br>
# years and months are your dictionaries<br>
<div class="im">years = {'1972':0,'1973':1,....}<br>
</div>months = {'Jan':0,'Feb':1,...,'Dec':11}<br>
data = open(your_data).readlines()<br>
for line in data:<br>
year,month,dat = line.split()<br>
y = int(('%('+year+')s') % years)<br>
m = int(('%('+month+')s') % months)<br>
rain_fall[y,m] = float(dat)<br>
<br>
[also - if someone knows how to use the dictionaries more appropriately here - please chime in]<br>
<br>
then you also have to think about what to do with the empty data sets. The initialization<br>
<br>
rain_fall = np.zeros([n_years,n_months])<br>
<br>
will have placed zeros everywhere - and if the data is missing - it won't get re-written. So that will make your analysis bogus also - so you have to walk through and replace the zeros with something else, like 'nan'. And then you could think about replacing missing data by averages - eg. replace a missing June entry by the average over all the non-zero June data.<br>
<br>
<br>
I was just hoping to give you a working example that you could use to make a functioning well thought out example that can handle the exceptions which will arise (like missing data, or a data file with a string where a float should be etc')<br>
<br>
<br>
Have fun!<br>
<span class="HOEnZb"><font color="#888888"><br>
Andre<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
<br>
<br>
On May 8, 2012, at 5:48 PM, questions anon wrote:<br>
<br>
> On Tue, May 8, 2012 at 4:41 PM, Andre' Walker-Loud <<a href="mailto:walksloud@gmail.com">walksloud@gmail.com</a>> wrote:<br>
> Hello anonymous questioner,<br>
><br>
> first comment - you may want to look into hdf5 data structures<br>
><br>
> <a href="http://www.hdfgroup.org/HDF5/" target="_blank">http://www.hdfgroup.org/HDF5/</a><br>
><br>
> and the python tools to play with them<br>
><br>
> pytables - <a href="http://www.pytables.org/moin" target="_blank">http://www.pytables.org/moin</a><br>
> h5py - <a href="http://code.google.com/p/h5py/" target="_blank">http://code.google.com/p/h5py/</a><br>
><br>
> I have personally used pytables more - but not for any good reason. If you happen to have the Enthought python distribution - these come with the package, as well as an installation of hdf5<br>
><br>
> hdf5 is a very nice file format for storing large amounts of data (binary) with descriptive meta-data. Also, numpy plays very nice with hdf5. Given all your questions here, I suspect you would benefit from learning about these and learning to play with them.<br>
><br>
> Now to your specific question.<br>
><br>
> > I would like to calculate summary statistics of rainfall based on year and month.<br>
> > I have the data in a text file (although could put in any format if it helps) extending over approx 40 years:<br>
> > YEAR MONTH MeanRain<br>
> > 1972 Jan 12.7083199<br>
> > 1972 Feb 14.17007142<br>
> > 1972 Mar 14.5659302<br>
> > 1972 Apr 1.508517302<br>
> > 1972 May 2.780009889<br>
> > 1972 Jun 1.609619287<br>
> > 1972 Jul 0.138150181<br>
> > 1972 Aug 0.214346148<br>
> > 1972 Sep 1.322102228<br>
> ><br>
> > I would like to be able to calculate the total rain annually:<br>
> ><br>
> > YEAR Annualrainfall<br>
> > 1972 400<br>
> > 1973 300<br>
> > 1974 350<br>
> > ....<br>
> > 2011 400<br>
> ><br>
> > and also the monthly mean rainfall for all years:<br>
> ><br>
> > YEAR MonthlyMeanRain<br>
> > Jan 13<br>
> > Feb 15<br>
> > Mar 8<br>
> > .....<br>
> > Dec 13<br>
> ><br>
> ><br>
> > Is this something I can easily do?<br>
><br>
> Yes - this should be very easy. Imagine importing all this data into a numpy array<br>
><br>
> ===<br>
> import numpy as np<br>
><br>
> data = open(your_data).readlines()<br>
> years = []<br>
> for line in data:<br>
> if line.split()[0] not in years:<br>
> years.append(line.split()[0])<br>
> months = ['Jan','Feb',....,'Dec']<br>
><br>
> rain_fall = np.zeros([len(n_year),len(months)])<br>
> for y,year in enumerate(years):<br>
> for m,month in enumerate(months):<br>
> rain_fall[y,m] = float(data[ y * 12 + m].split()[2])<br>
><br>
> # to get average per year - average over months - axis=1<br>
> print np.mean(rain_fall,axis=1)<br>
><br>
> # to get average per month - average over years - axis=0<br>
> print np.mean(rain_fall,axis=0)<br>
><br>
> ===<br>
><br>
> now you should imagine doing this by setting up dictionaries, so that you can request an average for year 1972 or for month March. That is why I used the enumerate function before to walk the indices - so that you can imagine building the dictionary simultaneously.<br>
><br>
> years = {'1972':0, '1973':1, ....}<br>
> months = {'Jan':0,'Feb':1,...'Dec':11}<br>
><br>
> then you can access and store the data to the array using these dictionaries.<br>
><br>
> print rain_fall[int('%(1984)s' % years), int('%(March)s' % months)]<br>
><br>
><br>
> Andre<br>
><br>
><br>
><br>
><br>
><br>
> > I have started by simply importing the text file but data is not represented as time so that is probably my first problem and then I am not sure how to group them by month/year.<br>
> ><br>
> > textfile=r"textfile.txt"<br>
> > f=np.genfromtxt(textfile,skip_header=1)<br>
> ><br>
> > Any feedback will be greatly appreciated.<br>
> ><br>
> > _______________________________________________<br>
> > Tutor maillist - <a href="mailto:Tutor@python.org">Tutor@python.org</a><br>
> > To unsubscribe or change subscription options:<br>
> > <a href="http://mail.python.org/mailman/listinfo/tutor" target="_blank">http://mail.python.org/mailman/listinfo/tutor</a><br>
><br>
><br>
<br>
</div></div></blockquote></div><br>