best way to read a huge ascii file.
Heli
hemla21 at gmail.com
Fri Nov 25 10:17:39 EST 2016
Hi,
I have a huge ascii file(40G) and I have around 100M lines. I read this file using :
f=np.loadtxt(os.path.join(dir,myfile),delimiter=None,skiprows=0)
x=f1[:,1]
y=f1[:,2]
z=f1[:,3]
id=f1[:,0]
I will need the x,y,z and id arrays later for interpolations. The problem is reading the file takes around 80 min while the interpolation only takes 15 mins.
I was wondering if there is a more optimized way to read the file that would reduce the time to read the input file?
I have the same problem when writing the output using np.savetxt.
Thank you in Advance for your help,
More information about the Python-list
mailing list