[Tutor] Processing rows from CSV

[mhw at doctors.net.uk]

Dear Tutors,

A rather general question, I'm afraid. I have found myself writing some python code to handle some CSV data, using the csv. DictReader that generates a dict for each row with the key as the column heading and the value in the file as the item. Most operations involve code of the form: (Apologies for incorrect caps)

For row in reader:
    If row['foo'] == 'something' :
        do this etc.

Typically I'm checking some form of consistency, or adding an element to the row based on something in the row.

I know about the existence of awk/ sed etc. Which could do some of this, but I ran into some problems with date manipulation, etc that they don't seem to handle very well.

I wanted to ask if anyone knew of anything similar, as I'm worried about re-inventing the wheel. One option would be to use (e.g.) sqlite and then use select/ insert commands, but personally I'd much rather write conditions in python than sql.

My other question was about how I do this more efficiently: I don't want to read the whole thing into memory at once, but (AFAIK) csv.DictReader doesn't support something like readline() to deliver it one at a time.

Any comments gratefully received.

Matt
Sent from my BlackBerry® wireless device