How pickle helps in reading huge files?
Irmen de Jong
irmen.NOSPAM at xs4all.nl
Wed Oct 16 17:09:31 EDT 2013
On 16-10-2013 23:04, Peter Cacioppi wrote:
> On Tuesday, October 15, 2013 11:55:26 PM UTC-7, Harsh Jha wrote:
>> I've a huge csv file and I want to read stuff from it again and again. Is it useful
>> to pickle it and keep and then unpickle it whenever I need to use that data? Is it
>> faster that accessing that file simply by opening it again and again? Please
>> explain, why?
>>
>>
>>
>> Thank you.
>
> Surprising no-one else mentioned a fairly typical pattern for this sort of situation
> - the compromise between "read from disk" and "read from memory" is "implement a
> cache".
...or: use memory mapped I/O. Just let the OS deal with the 'caching' of memory pages.
Irmen
More information about the Python-list
mailing list