[Numpy-discussion] Trying to read 500M txt file using numpy.genfromtxt within ipython shell

[Chao YUE]

Dear all,

I received a file from others which contains ~30 million lines and in size
of ~500M.
I try read it with numpy.genfromtxt in ipython interactive mode. Then
ipython crashed.
The data contains lat,lon,var1,year, the year ranges from 1001 to 2006.
Finally I want to write the
data to netcdf for separate years and feed them into the model. I guess I
need a better way to do this?
anyone would be any idea is highly appreciated.


lon,lat,year,area_burned
-180.0,65.0,1001,0
-180.0,65.0,1002,0
-180.0,65.0,1003,0
-180.0,65.0,1004,0
-180.0,65.0,1005,0
-180.0,65.0,1006,0
-180.0,65.0,1007,0

thanks and cheers,

Chao

-- 
***********************************************************************************
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
************************************************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20120320/f21e9b4b/attachment.html>