Tremendous slowdown due to garbage collection
aaron.watters at gmail.com
Mon Apr 14 22:27:42 CEST 2008
> A question often asked--and I am not a big a fan of these sorts of
> questions, but it is worth thinking about--of people who are creating
> very large data structures in Python is "Why are you doing that?"
> That is, you should consider whether some kind of database solution
> would be better. You mention lots of dicts--it sounds like some
> balanced B-trees with disk loading on demand could be a good choice.
Well, probably because you can get better
than 100x improved performance
if you don't involve the disk and use clever in memory indexing.
BTW, I think the default behaviour of the gc is
pretty silly. I tend to disable automatic gc and explicitly put in
collections when I know I'm done with some big operation these
-- Aaron Watters
More information about the Python-list