best way to read a huge ascii file.
BartC
bc at freeuk.com
Wed Nov 30 11:26:39 EST 2016
On 30/11/2016 16:16, Heli wrote:
> Hi all,
>
> Writing my ASCII file once to either of pickle or npy or hdf data types and then working afterwards on the result binary file reduced the read time from 80(min) to 2 seconds.
240,000% faster? Something doesn't sound quite right! How big is the
file now? (The one that had been 40GB of text and contained 100M lines
or datasets, although file caching is just about a possibility.)
--
Bartc
More information about the Python-list
mailing list