[Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)

[Andrew MacIntyre]

Mike Coleman wrote:
> Andrew, this is on an (intel) x86_64 box with 64GB of RAM.  I don't
> recall the maker or details of the architecture off the top of my
> head, but it would be something "off the rack" from Dell or maybe HP.
> There were other users on the box at the time, but nothing heavy or
> that gave me any reason to think was affecting my program.
> 
> It's running CentOS 5 I think, so that might make glibc several years
> old.  Your malloc idea sounds plausible to me.  If it is a libc
> problem, it would be nice if there was some way we could tell malloc
> to "live for today because there is no tomorrow" in the terminal phase
> of the program.
> 
> I'm not sure exactly how to attack this.  Callgrind is cool, but no
> way will work on something this size.  Timed ltrace output might be
> interesting.  Or maybe a gprof'ed Python, though that's more work.

Some malloc()s (notably FreeBSD's) can be externally tuned at runtime
via options in environment variables or other mechanisms - the malloc
man page on your system might be helpful if your platform has something
like this.

It is likely that PyMalloc would be better with a way to disable the
free()ing of empty arenas, or move to an arrangement where (like the
various type free-lists in 2.6+) explicit action can force pruning of
empty arenas - there are other usage patterns than yours which would
benefit (performance wise) from not freeing arenas automatically.

-- 
-------------------------------------------------------------------------
Andrew I MacIntyre                     "These thoughts are mine alone..."
E-mail: andymac at bullseye.apana.org.au  (pref) | Snail: PO Box 370
        andymac at pcug.org.au             (alt) |        Belconnen ACT 2616
Web:    http://www.andymac.org/               |        Australia