too much memory use

[Marc 'BlackJack' Rintsch]

In <1179860235.044137.295080 at a26g2000pre.googlegroups.com>, rohit wrote:

> i am making a program for desktop search.
> in order to make it more effective im implementing the records in the
> form of a tree(files starting with 'a','b','c'....have different
> trees ..so 26 trees in all) in memory and writing it down in file.
> the max size file has 16000 records...now to implement the tree in
> list i'm using line no as index ..and empty child nodes are
> represented as "\n"
> all this work is going on in the memory..
> problem is the system eats up my 512 mb RAM +1gb virtual store n hangs
> cant think of an effective way to implement tree in memory(i can
> compact it on disk by writing just the index no..along with the record
> from which tree in memory can be reconstructed, but i have to
> implement tree as previous to implement random access)

I'm not quite sure what exactly you have as in-memory data structures and
how many "records" -- are you sure you don't keep references to objects
you don't really need anymore?  Or maybe you have object cycles and
implemented the `__del__()` method on those objects?

Anyway… If the data doesn't fit into memory anymore it's time to put
them into a database.  Either a `shelve`, an SQL database like SQLite or
maybe an object database like zodb or Durus.

Ciao,
	Marc 'BlackJack' Rintsch