pickle performance on larger objects

[Sam Penrose]

Doh! That's what I get for reading the docs too quickly.

With bin = 1:

 >>> time_cPickle_Load()
dumping list of 10 dicts:
dumped: 0.00358998775482
loading list of 10 dicts:
loaded: 0.00386297702789
dumping list of 100 dicts:
dumped: 0.0141079425812
loading list of 100 dicts:
loaded: 0.0700759887695
dumping list of 1000 dicts:
dumped: 0.188324928284 # ~ 2x speedup
loading list of 1000 dicts:
loaded: 0.379488945007   # ~ 20x speedup
dumping list of 10000 dicts:
dumped: 2.23442697525   # ~ 2x speedup
loading list of 10000 dicts:
loaded: 3.07298398018     #  ~ 40x speedup

memory usage increases by about 20%, FWIW. For my particular use case  
cPickle is still several (many ?) times slower than just recreating the 
object by reading in a file. What implications this has for best 
practices in persistence of larger objects I do not know, but I hope the 
data point is of interest to others.

On Wednesday, July 17, 2002, at 02:13 PM, Geoffrey Talvola wrote:

> Try using binary pickle format (pass in 1 as an extra third argument to
> dump()).  It's quite a bit faster both dumping and loading.
>
> - Geoff
>
>> -----Original Message-----
>> From: Sam Penrose [mailto:spenrose at intersight.com]
>> Sent: Wednesday, July 17, 2002 5:09 PM
>> To: python-list at python.org
>> Subject: pickle performance on larger objects
>>
>>
>> On a recent project we decided to use pickle for some quick-and-dirty
>> object persistence. The object in question is a list of 3,000
>> dictionaries
>> whose keys and values are short (< 100 character) strings--about 1.5
>> megs worth of character data in total. Loading this object
>> from a pickle
>> using cPickle took so long we assumed something was broken.
>>
>