Dictionaries as records

[Aahz Maruch]

In article <QKQT7.27703$t07.3920324 at twister.midsouth.rr.com>,
Bill Wilkinson <bwilk_97 at yahoo.com> wrote:
>
>I have been happily using a list of dictionaries to hold table data for
>years.  For the first time, this method is proving less than efficient
>because of the amount of memory overhead the dictionaries produce.  I
>have a file with 200K records and 16 fields.  This file is parsed and
>each row is put into a dictionary and the dictionary is added to a
>list.  The raw file is only about 50mb.
>
>I was shocked to see that my memory use jumped to 500MB!  When I delete
>the list the memory is returned to the system, so I know that the
>memory is being used in the dictionaries.
>
>What strikes me as odd is that I can create a list of 200K dictionaries
>with test data (a copy of the same record over and over) and the amount
>of memory used is only half.

My guess is that you're creating the dict keys dynamically at runtime.
Try either hardcoding the dict keys or interning the strings you're
reading in.
-- 
                      --- Aahz  <*>  (Copyright 2001 by aahz at pobox.com)

Hugs and backrubs -- I break Rule 6                 http://www.rahul.net/aahz/
Androgynous poly kinky vanilla queer het Pythonista   

Tenth Virtual Anniversary: 10 days and counting