[Tutor] Processing CSV files

[Alan Gauld]

On 08/10/13 21:46, Leena Gupta wrote:

> I need to process a large csv file every 5-10 minutes. The file could
> contain 3mill to 10 mill rows and size could be 6MB to 10MB(+).

Assuming you have a modern multi-core PC with >2G RAM then those sizes 
should not be a problem. If you have an older PC then you may need to be 
a bit more clever about how you load the files.

> ... I need to sum up a number value by grouping on
> certain attributes and store the output in a datastore.

Again that sounds fairly normal. There are modules that can
help with that too.

> I wanted to know if Python is recommended and can it be used
 > for processing data in csv files of this size?

Yes, for sure.

> Python has a csv library as well.

Yes and it is quite well documented but if you have specific
questions we can try to help.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.flickr.com/photos/alangauldphotos