Memory error while saving dictionary of size 65000X50 using pickle

Larry Bates larry.bates at websafe.com`
Mon Jul 7 21:13:28 CEST 2008


Nagu wrote:
> I am trying to save a dictionary of size 65000X50 to a local file and
> I get the memory error problem.
> 
> How do I go about resolving this? Is there way to partition the pickle
> object and combine later if this is a problem due to limited resources
> (memory) on the machine (it is 32 bit machine Win XP, with 4GB RAM).
> 
> 
> Here is the detail description of the error:
> 
> Traceback (most recent call last):
>   File "<pyshell#12>", line 1, in <module>
>     s = pickle.dumps(itemsim)
>   File "C:\Python25\lib\pickle.py", line 1366, in dumps
>     Pickler(file, protocol).dump(obj)
>   File "C:\Python25\lib\pickle.py", line 224, in dump
>     self.save(obj)
>   File "C:\Python25\lib\pickle.py", line 286, in save
>     f(self, obj) # Call unbound method with explicit self
>   File "C:\Python25\lib\pickle.py", line 649, in save_dict
>     self._batch_setitems(obj.iteritems())
>   File "C:\Python25\lib\pickle.py", line 663, in _batch_setitems
>     save(v)
>   File "C:\Python25\lib\pickle.py", line 286, in save
>     f(self, obj) # Call unbound method with explicit self
>   File "C:\Python25\lib\pickle.py", line 600, in save_list
>     self._batch_appends(iter(obj))
>   File "C:\Python25\lib\pickle.py", line 615, in _batch_appends
>     save(x)
>   File "C:\Python25\lib\pickle.py", line 286, in save
>     f(self, obj) # Call unbound method with explicit self
>   File "C:\Python25\lib\pickle.py", line 562, in save_tuple
>     save(element)
>   File "C:\Python25\lib\pickle.py", line 286, in save
>     f(self, obj) # Call unbound method with explicit self
>   File "C:\Python25\lib\pickle.py", line 477, in save_float
>     self.write(FLOAT + repr(obj) + '\n')
> MemoryError: out of memory

I've generated some rather large pickled dicts before and have never seen this 
before.  You can, of course, split your dictionary into several smaller ones, 
pickle them individually and combine them back together on unpickle/read using 
the dictionary update method.

BTW - 32-bit Windows can only address 3.5Gb of memory maximum.  If you have more 
installed, it is ignored.

-Larry



More information about the Python-list mailing list