Pop a list from beginning ? and memory saving...

[Chris Liechti]

"Shagshag13" <shagshag13 at yahoo.fr> wrote in
news:add289$vud42$1 at ID-146704.news.dfncis.de: 
> and yes i'm working with large amount of data : i'm trying to index
> something like 1 go of plain text by using conventionnal text

1go thats 1GB of data for those who don't know french :-)

> retrieval structures (i mean without compression tricks, because i
> *don't* have time to implement theses, but with stoplist and so on) 
> and wishing to do it in one pass without relying on hdd temp files
> (i'm starting to think that i were dreaming...)
> 
> the most i think i could simply do is this : i use integer each time
> as possible, i free everything i don't need anymore with = None, and
> what could i do more ? Advices are ***greatly*** welcome !!!

- use "del var" instead of "= None".
- try to work in junks, do you realy hae 1GB in memory? maybe you can work 
in blocks of 1MB of 64KB or so.
- if the result is big and not immediatly needed in the program, then write 
it to the disk.
- maybe two-pass isn't that bad - especialy if your first step algorithm 
reduces the amount of data by some factors.

chris

-- 
Chris <cliechti at gmx.net>