[Tutor] data analysis with python

[Andre' Walker-Loud]

Hi David,

> I'm trying to use python for analysing data from building energy simulations and was wondering whether there is way to do this without using anything sql like. 
> 
> The simulations are typically run for a full year, every hour, i.e. there are 8760 rows and about 100+ variables such as external air temperature, internal air temperature, humidity, heating load, ... making roughly a million data points. I've got the data in a csv file and also managed to write it in a sqlite db.
> 
> I would like to make requests like the following:
> 
> Show the number of hours the aircon is running at 10%, 20%, ..., 100%
> Show me the average, min, max air temperature, humidity, solar gains,.... when the aircon is running at 10%, 20%,...,100%
> 
> Eventually I'd also like to generate an automated html or pdf report with graphs. Creating graphs is actually somewhat essential.
> 
> I tried sql  and find it horrible, error prone, too much to write, the logic somehow seems to work different than my brain and I couldn't find particulary good documentation (particulary the documentation of the api is terrible, in my humble opinion). I heard about zope db which might be an alternative. Would you mind pointing me towards an appropriate way to solve my problem? Is there a way for me to avoid having to learn sql or am I doomed?


I would recommend learning 

hdf5	http://www.hdfgroup.org/HDF5/

and the python utility to interface with it

pytables		http://www.pytables.org/moin


and numpy and scipy are great for data analysis (python libraries) - numpy handles things like linear algebra, scipy has many built in scientific functions.  And then matplotlib for plotting (very similar functions to matlab if you are familiar with it).  Lastly, a very nice interface is "iPython", which is basically an enhanced python interpreter designed for/by science types.

All of these tools are installed for you with the Enthought Python Distribution (full dist is free if you have a .edu address, otherwise they provide a light version with basic libraries, and you can install others you like)

http://www.enthought.com/


If you have any specific questions on these (I know that is a lot to look into right away) let me know.


Cheers,

Andre