optimizing memory usage of large in memory structures

[Hugo Liu]

hi,

i'm hoping that someone out there has a solution to hack this problem.

working with corpus analysis and large knowledge bases, i often have
to load large arrays into memory.  each array element is just a text
string.  although the text is only 5mb, python consumes 40+ mb.  i
would suspect that the overhead is from storing type info for each
element.  if the strings were all fixed length, i'd just store them
ordered and fixed width into a single string and use bisect, but they
aren't fixed width.

does anyone know if there is a way to fake static typing for this
situation or any other way to make this more efficient (without
sacrificing operating system interoperability or having to write
external C code?)  any suggestions or pointers would be much
appreciated!

hugo