Creating daily/monthly averages from datafiles

[Fernando Pérez]

At the risk of sounding stupid, may I suggest you look into Numeric for this? 
It has most of what you want already in it, and it's fast to boot. Multi-year 
data every 15 minutes is bound to be a lot, I can't imagine how slow doing 
that in raw python will be. Unless you are going to throw a 1024 node cluster 
at this simple problem :)

Now, the trick with Numeric is that it wants homogeneous data, so none of that 
date/data mixing you want. But with a bit of helper tables, at some extra 
storage cost, I'm willing to bet you can fix this very easily. Store the 
numerical data in one or a few Numeric arrays, and have a few auxiliary 
python structures (dicts, probably) to help along with the indexing. This way 
you can quickly find where in the Numarray the data you want is, and then you 
can efficiently process it using Numeric.

Just my $0.02.

Cheers,

f.