Pop a list from beginning ? and memory saving...
cliechti at gmx.net
Sun Jun 2 20:13:53 CEST 2002
"Shagshag13" <shagshag13 at yahoo.fr> wrote in
news:add289$vud42$1 at ID-146704.news.dfncis.de:
> and yes i'm working with large amount of data : i'm trying to index
> something like 1 go of plain text by using conventionnal text
1go thats 1GB of data for those who don't know french :-)
> retrieval structures (i mean without compression tricks, because i
> *don't* have time to implement theses, but with stoplist and so on)
> and wishing to do it in one pass without relying on hdd temp files
> (i'm starting to think that i were dreaming...)
> the most i think i could simply do is this : i use integer each time
> as possible, i free everything i don't need anymore with = None, and
> what could i do more ? Advices are ***greatly*** welcome !!!
- use "del var" instead of "= None".
- try to work in junks, do you realy hae 1GB in memory? maybe you can work
in blocks of 1MB of 64KB or so.
- if the result is big and not immediatly needed in the program, then write
it to the disk.
- maybe two-pass isn't that bad - especialy if your first step algorithm
reduces the amount of data by some factors.
Chris <cliechti at gmx.net>
More information about the Python-list