minidom memory performance

[Peter Hansen]

Geoff Gerrietts wrote:
> 
> I was noticing today -- after some talks with a friend who tried to
> use minidom on a 28MB XML file and having it run thru 8GB of main
> memory on Alpha -- that when I load my own 26kB XML file into a
> minidom tree, my process size grows a little more than 2MB.
> 
> By my very rough calculations, that's in the neighborhood of 10000%
> memory consumption. Does this seem right to everyone? Is this just the
> overhead that goes with using pure python classes to implement a
> data model that's heavy on classes and light on data? Is there
> something else that's going on there that I don't understand?

I'd say yes.  Anyway, it's very unlikely that using the DOM on a 28MB
XML file is a good idea.  Using SAX might be more effective, or a 
much lighter-weight tree-based approach like PyRXP.  What's in this
massive file?  Can it just be processed chunk by chunk?

-Peter