[Python-Dev] RE: [Zope.Com Geeks] Re: Program very slow to finish

[Tim Peters]

[Barry]
> Very interesting!  I think I've noticed the same thing with some
> Pipermail (Mailman's archiver) work I've been doing.  When doing a
> from-scratch re-generation of the archives of say, python-list
> (280+MB), it creates some really big dicts, and when it goes to free
> them (apparently in gc), the disk just gets hammered.  Relatively
> little CPU is being used (it's like 80-90% idle), but the machine is
> almost unresponsive.  This on a 2.4.2 kernel with 256MB.

Maybe related, but not the same:  Roeland was careful to run tests small
enough that the disk didn't get involved.  His slowdowns were on all-in-RAM
cases, and he reported high CPU utilization throughout.

Once the disk gets involved, it can be a true nightmare:  while we traverse
the dict in cache-friendly left-to-right sequential order, the memory it
points *at* jumps all over creation from one slot to the next.  Several
years ago I amusedly waited 3 hours for a Python program to terminate on my
old Win95 box; I would have waited longer except the disk grinding was
interfering with hearing the TV <wink>.

We could likely "fix that", via sorting the dict entries by pointed-at
memory address before decref'ing (waste a second to save a day); note that
there's already a superstitious feeble approximation to that in
list_dealloc.

> I'll try building a 2.2 with pymalloc and see what happens.

Well, I'm waiting <wink>.