[Numpy-discussion] loading data

Mag Gam magawake at gmail.com
Thu Jun 25 20:59:06 EDT 2009


I am very new to NumPy and Python. We are doing some research in our
Physics lab and we need to store massive amounts of data (100GB
daily). I therefore, am going to use hdf5 and h5py. The problem is I
am using np.loadtxt() to create my array and create a dataset
according to that. np.loadtxt() is reading a file which is about 50GB.
This takes a very long time! I was wondering if there was a much
easier and better way of doing this.


More information about the NumPy-Discussion mailing list