optimizing memory usage of large in memory structures

[Michael Hudson]

theory79 at yahoo.com (Hugo Liu) writes:

> hi,
> 
> i'm hoping that someone out there has a solution to hack this problem.
> 
> working with corpus analysis and large knowledge bases, i often have
> to load large arrays into memory.  each array element is just a text
> string.  although the text is only 5mb, python consumes 40+ mb.  i
> would suspect that the overhead is from storing type info for each
> element.  if the strings were all fixed length, i'd just store them
> ordered and fixed width into a single string and use bisect, but they
> aren't fixed width.
> 
> does anyone know if there is a way to fake static typing for this
> situation

Don't think so.  You could try using arrays, but I doubt that would
save you much.

> or any other way to make this more efficient (without
> sacrificing operating system interoperability or having to write
> external C code?)  any suggestions or pointers would be much
> appreciated!

The only thing that leaps to mind is having one huge string and
instead of passing strings around, pass (start, end) tuples.  But
that may just be a crazy idea...

Cheers,
M.

-- 
  Hey, if I thought I was wrong, I'd change my mind.  :)
                                    -- Grant Edwards, comp.lang.python