[Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)

Mike Coleman tutufan at gmail.com
Sun Dec 21 01:05:19 CET 2008


Tim, I left out some details that I believe probably rule out the
"swapped out" theory.  The machine in question has 64GB RAM, but only
16GB swap.  I'd prefer more swap, but in any case only around ~400MB
of the swap was actually in use during my program's entire run.
Furthermore, during my program's exit, it was using 100% CPU, and I'm
95% sure there was no significant "system" or "wait" CPU time for the
system.  (All observations via 'top'.)  So, I think that the problem
is entirely a computational one within this process.

The system does have 8 CPUs.  I'm not sure about it's memory
architecture, but if it's some kind of NUMA box, I guess access to
memory could be slower than what we'd normally expect.  I'm skeptical
about that being a significant factor here, though.

Just to clarify, I didn't gc.disable() to address this problem, but
rather because it destroys performance during the creation of the huge
dict.  I don't have a specific number, but I think disabling gc
reduced construction from something like 70 minutes to 5 (or maybe
10).  Quite dramatic.

Mike


>From Tim Peters:
BTW, the original poster should try this:  use whatever tools the OS
supplies to look at CPU and disk usage during the long exit.  What I
/expect/ is that almost no CPU time is being used, while the disk is
grinding itself to dust.  That's what happens when a large number of
objects have been swapped out to disk, and exit processing has to page
them all back into memory again (in order to decrement their
refcounts).


More information about the Python-Dev mailing list