[Numpy-discussion] loading data

[Mag Gam]

Hello.

I am very new to NumPy and Python. We are doing some research in our
Physics lab and we need to store massive amounts of data (100GB
daily). I therefore, am going to use hdf5 and h5py. The problem is I
am using np.loadtxt() to create my array and create a dataset
according to that. np.loadtxt() is reading a file which is about 50GB.
This takes a very long time! I was wondering if there was a much
easier and better way of doing this.

TIA