steve at holdenweb.com
Wed Mar 21 18:49:35 CET 2007
Tom Wright wrote:
> Steven D'Aprano wrote:
>> You've described an extremely artificial set of circumstances: you create
>> 40,000,000 distinct integers, then immediately destroy them. The obvious
>> solution to that "problem" of Python caching millions of integers you
>> don't need is not to create them in the first place.
> I know it's a very artificial setup - I was trying to make the situation
> simple to demonstrate in a few lines. The point was that it's not caching
> the values of those integers, as they can never be read again through the
> Python interface. It's just holding onto the space they occupy in case
> it's needed again.
>> So what's your actual problem that you are trying to solve?
> I have a program which reads a few thousand text files, converts each to a
> list (with readlines()), creates a short summary of the contents of each (a
> few floating point numbers) and stores this summary in a master list. From
> the amount of memory it's using, I think that the lists containing the
> contents of each file are kept in memory, even after there are no
> references to them. Also, if I tell it to discard the master list and
> re-read all the files, the memory use nearly doubles so I presume it's
> keeping the lot in memory.
I'd like to bet you are keeping references to them without realizing it.
The interpreter won't generally allocate memory that it can get by
garbage collection, and reference counting pretty much eliminates the
need for garbage collection anyway except when you create cyclic data
> The program may run through several collections of files, but it only keeps
> a reference to the master list of the most recent collection it's looked
> at. Obviously, it's not ideal if all the old collections hang around too,
> taking up space and causing the machine to swap.
We may need to see code here for you to convince us of the correctness
of your hypothesis. It sounds pretty screwy to me.
>>> but is there anything I can do to get that memory back without closing
>> Why do you want to manage memory yourself anyway? It seems like a
>> horrible, horrible waste to use a language designed to manage memory for
>> you, then insist on over-riding it's memory management.
> I agree. I don't want to manage it myself. I just want it to re-use memory
> or hand it back to the OS if it's got an awful lot that it's not using.
> Wouldn't you say it was wasteful if (say) an image editor kept an
> uncompressed copy of an image around in memory after the image had been
Yes, but I'd say it was the programmer's fault if it turned out that the
interpreter wasn't doing anything wrong ;-) It could be something inside
an exception handler that is keeping a reference to a stack frame or
something silly like that.
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
Recent Ramblings http://holdenweb.blogspot.com
More information about the Python-list