Python dictionary size/entry limit?

[Tino Wildenhain]

intelliminer at gmail.com wrote:
> I wrote a script to process textual data and extract phrases from
> them, storing these phrases in a dictionary. It encounters a
> MemoryError when there are about 11.18M keys in the dictionary, and
> the size is about 1.5GB. I tried multiple times, and the error occurs
> everytime at exactly the same place (with the same number of keys in
> the dict). I then split the dictionary into two using a simple
> algorithm:
> 
> if str[0]<='m':
>     dict=dict1
> else:
>     dict=dict2
> 
> #use dict...
> 
> And it worked fine. The total size of the two dictionaries well
> exceeded 2GB yet no MemoryError occured.
> 
> I have 1GB of pysical memory and 3GB in pagefile. Is there a limit to
> the size or number of entries that a single dictionary can possess? By
> searching on the web I can't find a clue why this problem occurs.

 From what can be deducted from the headers of your message:
X-HTTP-UserAgent: Mozilla/5.0 (Windows; U; Windows NT 5.1;...
you are using windows?
It seems either python or windows memory management somehow prevent
the use of continuous memory areas that large.
We've got such an example somewhere down the list which was similar
(iirc it was a large string in memory) which runned perfectly
with linux. You can try yourself maybe by installing ubuntu
on the same host. (If you feel fit you can even skip the install
and run it off life CD but then you need to fiddle a little to
get swap space on disk)

Regards
Tino



> --
> http://mail.python.org/mailman/listinfo/python-list

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3241 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://mail.python.org/pipermail/python-list/attachments/20090221/5cb8f602/attachment.bin>