Cache a large list to disk

[Svein Ove Aas]

Peter Otten wrote:

> Chris wrote:
> 
>> week, so I can safely cache this data to a big file on disk, and read
>> out of this big file -- rather than having to read about 10,000 files
>> -- when the program is loaded.
>> 
>> Now, if this were C I'd know how to do this in a pretty
>> straightforward manner.  But being new to Python, I don't know how I
>> can (hopefully easily) write this data to a file, and then read it out
>> into memory on subsequent launches.
> 
> Have a look at pickle:
> 
>>>> data = [{'el2': 0, 'el3': 0, 'el1': 0, 'el4': 0, 'el5': 0},
> ...  {'el2': 15, 'el3': 21, 'el1': 9, 'el4': 33, 'el5': 51},
> ...  {'el2': 35, 'el3': 49, 'el1': 21, 'el4': 77, 'el5': 119},
> ...  {'el2': 45, 'el3': 63, 'el1': 27, 'el4': 99, 'el5': 153}]
>>>>
>>>> import cPickle as pickle # cPickle is pickle implemented in C

And, yes, cPickle is faster. A lot faster.

There are switches you can throw to have it use binary instead of sticking
to readable characters for some savings, too.