[Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)

[Mike Coleman]

On Mon, Dec 22, 2008 at 2:54 PM, Ivan Krstić
<krstic at solarsail.hcs.harvard.edu> wrote:
> It's still not clear to me, from reading the whole thread, precisely what
> you're seeing. A self-contained test case, preferably with generated random
> data, would be great, and save everyone a lot of investigation time.

I'm still working on a test case.  The first couple of attempts, using
a half-hearted attempt to model the application behavior wrt this dict
didn't demonstrate bad behavior.

My impression is that no one's burning much time on this but me at the
moment, aside from offering helpful advice.  If you are, you might
want to wait.  I noticed just now that the original hardware was
throwing some chipkills, so I'm retesting on something else.


> In the
> meantime, can you 1) turn off all swap files and partitions, and 2) confirm
> positively that your CPU cycles are burning up in userland?

For (1), I don't have that much control over the machine.  Plus, based
on watching with top, I seriously doubt the process is using swap in
any way.  For (2), yes, 100% CPU usage.

> (In general, unless you know exactly why your workload needs swap, and have
> written your program to take swapping into account, having _any_ swap on a
> machine with 64GB RAM is lunacy. The machine will grind to a complete
> standstill long before filling up gigabytes of swap.)

The swap is not there to support my application per se.  Clearly if
you're swapping, generally you're crawling.  This host is used by a
reasonably large set of non- and novice programmers, who sometimes
vacuum up VM without realizing it.  If you have a nice, big swap
space, you can 'kill -STOP' these offenders, and allow them to swap
out while you have a leisurely discussion with the owner and possibly
'kill -CONT' later, as opposed to having to do a quick 'kill -KILL' to
save the machine.  That's my thinking, anyway.

Mike