[Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)

[Tim Peters]

[M.-A. Lemburg]
>> These long exit times are usually caused by the garbage collection
>> of objects. This can be a very time consuming task.

[Leif Walsh]
> In that case, the question would be "why is the interpreter collecting
> garbage when it knows we're trying to exit anyway?".

Because user-defined destructors (like __del__ methods and weakref
callbacks) may be associated with garbage, and users presumably want
those to execute.  Doing so requires identifying identifying garbage
and releasing it, same as if the interpreter didn't happen to be
exiting.

BTW, the original poster should try this:  use whatever tools the OS
supplies to look at CPU and disk usage during the long exit.  What I
/expect/ is that almost no CPU time is being used, while the disk is
grinding itself to dust.  That's what happens when a large number of
objects have been swapped out to disk, and exit processing has to page
them all back into memory again (in order to decrement their
refcounts).  Python's cyclic gc (the `gc` module) has nothing to do
with this -- it's typically the been-there-forever refcount-based
non-cyclic gc that accounts for supernaturally long exit times.

If that is the case here, there's no evident general solution.  If you
have millions of objects still alive at exit, refcount-based
reclamation has to visit all of them, and if they've been swapped out
to disk it can take a very long time to swap them all back into memory
again.