Memory limit to dict?

Peter Beattie Peter.Beattie at web.de
Tue Apr 11 13:45:19 EDT 2006


I was wondering whether certain data structures in Python, e.g. dict,
might have limits as to the amount of memory they're allowed to take up.
Is there any documentation on that?

Why am I asking? I'm reading 3.6 GB worth of BLAST output files into a
nested dictionary structure (dict within dict ...). Looks something like
this:

{ GenomeID:
  { ProteinID:
    { GenomeID:
      { ProteinID, Score, PercentValue, EValue } } } }

Now, the thing is: Even on a machine with 16 GB RAM, the program
terminates with a MemoryError, obviously long before the machine's RAM
is used up.

I've no idea how far the Windows task manager's resource monitor can be
trusted -- probably not as far as I could throw a heavy-set St Bernard
--, but it seems to stop roughly when that monitor records a swap file
size of 2.2 GB.

Barring any revamping of the code itself, which I will have to do
anyway, is there anything so far that would indicate a problem inherent
to Python?

(I can post the relevant code too, of course, if that would help.)

TIA!

-- 
Peter



More information about the Python-list mailing list