[Tutor] Processing CSV files

[Mark Lawrence]

On 08/10/2013 21:46, Leena Gupta wrote:
> Hello,
>
> Looking for some inputs on Python's csv processing feature.
>
> I need to process a large csv file every 5-10 minutes. The file could
> contain 3mill to 10 mill rows and size could be 6MB to 10MB(+). As part
> of the processing, I need to sum up a number value by grouping on
> certain attributes and store the output in a datastore. I wanted to know
> if Python is recommended and can it be used for processing data in csv
> files of this size? Any issues that we need to be aware of? I believe
> Python has a csv library as well.
>
> Thanks!
>

1) As others have already stated Python can easily handle this.

2) I'd recommend using a csv.DictReader as it makes your code easier to 
read, please see 
http://docs.python.org/3/library/csv.html#csv.DictReader.  Note the 
fieldnames parameter is very useful if your files don't have header rows.

-- 
Roses are red,
Violets are blue,
Most poems rhyme,
But this one doesn't.

Mark Lawrence