best way to read a huge ascii file.
Chris Angelico
rosuav at gmail.com
Wed Nov 30 11:32:57 EST 2016
On Thu, Dec 1, 2016 at 3:26 AM, BartC <bc at freeuk.com> wrote:
> On 30/11/2016 16:16, Heli wrote:
>>
>> Hi all,
>>
>> Writing my ASCII file once to either of pickle or npy or hdf data types
>> and then working afterwards on the result binary file reduced the read time
>> from 80(min) to 2 seconds.
>
>
> 240,000% faster? Something doesn't sound quite right! How big is the file
> now? (The one that had been 40GB of text and contained 100M lines or
> datasets, although file caching is just about a possibility.)
Seems reasonable to me. Coming straight off the disk cache.
ChrisA
More information about the Python-list
mailing list