[Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)

[Steve Holden]

Antoine Pitrou wrote:
> Leif Walsh <leif.walsh <at> gmail.com> writes:
>> It might be a semantic change that I'm looking for here, but it seems
>> to me that if you turn off the garbage collector, you should be able
>> to expect that either it also won't run on exit, or it should have a
>> way of letting you tell it not to run on exit. 
> [...]
> 
> I'm skeptical that it's a garbage collector problem. The script creates one dict
> containing lots of strings and ints. The thing is, strings and ints aren't
> tracked by the GC as they are simple atomic objects. Therefore, the /only/
> object created by the script which is tracked by the GC is the dict. Moreover,
> since there is no cycle created, the dict should be directly destroyed when its
> last reference dies (the "del" statement), not go through the garbage collection
> process.
> 
> Given that the problem is reproduced on certain systems and not others, it can
> be related to an interaction between allocation patterns of the dict
> implementation, the Python memory allocator, and the implementation of the C
> malloc() / free() functions. I'm no expert enough to find out more on the
> subject.
> 
I believe the OP engendered a certain amount of confusion by describing
object deallocation as being performed by the garbage collector. So he
perhaps didn't understand that even decref'ing all the objects only
referenced by the dict will take a huge amount of time unless there's
enough real memory to hold it.

regards
 Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/