[Tutor] Reading Multiple Files in Sequence

[Hansen, Mike]

> [mailto:tutor-bounces at python.org] On Behalf Of Kelvin Gorospe
> 
> Sorry, I need to stop doing this... Attached is the script 
> that I forgot to attach in my last email to the list.  Thanks again, K
> 
> 
> On Fri, Apr 18, 2008 at 11:22 AM, Kelvin Gorospe 
> <kdgorospe at gmail.com> wrote:
> 
> 
> 	Hi everyone,
> 	
> 	I'm new to Python AND programming and would appreciate 
> any help anybody has to offer. I have several .csv files that 
> contains temperature data taken at several different 
> locations.  Each line in the csv file is a list such as this: 
> timestamp at location 1, temperature at location 1, timestamp 
> at location 2, temperature at location 2, etc. 
> 	
> 	The attached script opens one file and creates an 
> output file that lists all the timestamps and temperatures 
> for location 1.  However, what I'd like to do, is have a 
> script that SEVERAL csv files (each one containing one month 
> of temperature data) and create an output file for EACH 
> location (approximately 50 of them).  So in the end, I'll 
> have 50 txt files showing time stamps and temperatures for a 
> single location over all months of data collection.
> 	
> 	Thanks in advance for your help.
> 	
> 	-Kelvin

Hmmmm..... first, I'd use the CSV module instead of manually parsing
the csv files. Although it might be ok in your application, sometimes
you can get killed by CSV files that have commas as part of the data.

Example: 

zone 1, 90, "Hansen, Mike", "555 Elm Street, MayBerry", 555-5555

Just splitting on commas will give you unexpected results since it
will catch the stuff between the quotes. The CSV module handles this.

Second, if you want to grab all csv files in a directory, you can use
the glob module. Then you can iterate and read each csv file.

Finally, the tricky part. There are a few ways of collecting the data
for each location. If there isn't a huge amount of data, then you
could load it into a in-memory data structure like a dictionary with
each key being the location and the value being a list of tuples or a
list of class instances, or a dictionary of dictionaries.

Example of dictionary with a list of tuples:
{ "location1" : [ (200804181611, 55), (200804171625, 42) , ...], 
  "location2" : [ (200804150800, 35)] }

Then you could iterate over each dictionary key and write out your
text file for that location.

Another way would be to load the data into a database. Python 2.5
comes with SQLite. You could create one database table with columns
for location, time, and temp. Then you can query the database for each
location.

I wouldn't open a file for each location while parsing the CSV files.
To me it seems ugly to have up to 50 files open at once.

I'm sure there are other ways that some of the more experienced tutor
list members can suggest.

Mike