[Tutor] Most efficient way to read large csv files with properly converted mixed data types.

[Alan Gauld]

On 26/06/16 07:21, Ek Esawi wrote:
> The first step of my plan is to do basic statistical analysis. The 2nd step
> is to chose a sample from each file and do more advanced statistical
> analysis for which i plan to use  R.

If you are just extracting data and repackaging it for R then the format
doesn't matter too much. You certainly won;t need a named tuple. Either
use the dictionary reader if you want to access fields by name
(especially good if the incoming CSV file format is likely to change) or
use the standard reader and use indexes to get the fields you need.

Writing it out you can again use the csv writers. Which writer will
depend on the format your stats package wants/prefers.


-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos