[Expat-bugs] [ expat-Bugs-632146 ] xmlwf: memory demands

SourceForge.net noreply at sourceforge.net
Thu Jan 23 13:00:39 EST 2003


Bugs item #632146, was opened at 2002-11-01 12:15
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=110127&aid=632146&group_id=10127

>Category: Documentation
Group: Not a Bug
Status: Open
Resolution: None
Priority: 5
Submitted By: Rolf Ade (pointsman)
Assigned to: Fred L. Drake, Jr. (fdrake)
Summary: xmlwf: memory demands

Initial Comment:

expat 1.95.5

Running xmlwf against bigger XML files needs almost the
double of the file  size of memory (at the end, the
process size grows permanently during the  processing).
This looks pretty wrong to me, since the tool does SAX
parsing.

Please notice, that this seems to be a problem only
with the xmlwf code, not with the expat lib per se. An
application by me, build on top of expat, doesn't shows
this behavior. Instead, the memory size of the process
stays constant and pretty low, even while parsing big
XML files - which is, what I also would have expected
from the xmlwf tool.

I'm sorry, for only reporting the problem and not
digging into.

rolf


----------------------------------------------------------------------

>Comment By: Fred L. Drake, Jr. (fdrake)
Date: 2003-01-23 16:00

Message:
Logged In: YES 
user_id=3066

This needs documentation, at least.  I don't see any
problems with the mmap code in xmlwf/unixfilemap.c.

----------------------------------------------------------------------

Comment By: Karl Waclawek (kwaclaw)
Date: 2002-11-20 19:10

Message:
Logged In: YES 
user_id=290026

As it turns out - after reading the manual - the default
behaviour of xmlwf is to use memory mapped files.
This explains the observed use of memory.

With the -r option xmlwf can be made to use
regular file I/O, and then memory use is as expected.

Maybe the question could be asked if xmlwf
should have a different default behaviour.
But then it would not be backwards compatible.

Leave open for Fred to comment.

----------------------------------------------------------------------

Comment By: Karl Waclawek (kwaclaw)
Date: 2002-11-20 13:52

Message:
Logged In: YES 
user_id=290026

The files you used for testing:

- Do they have an internal and/or external DTD subset?
- Are there external or internal entity references?
- Are there attributes on some/all elements?

----------------------------------------------------------------------

Comment By: Rolf Ade (pointsman)
Date: 2002-11-20 13:25

Message:
Logged In: YES 
user_id=13222

The problem shows even with no options given to xmlwf. i.e.
xmlwf  <xmlFile>

Just use a (big)  XML file and watch the xmlwf process grow
in the process table. (You need a really big one (several 10
MBytes) to have a change to see it with ordinary ps (on
unix) or the task manager (on MS), because otherwise expat
is finished, before you had a chance to see anything ;.)).

No, I would bet, it's not a (classical)  memory leak. I've
memory debugged expat based applications several times, and
the lib doesn't leak memory (or only very seldom under rare
circumstances,)

I've just valgrind'ed (mem leak checker) an xmlwf run, which
shows the depicted behavior (xmlwf process size grows far to
much), but there is no mem leak -. no pointer to allocated
memory is lost, and all allocated memory is cleand up at the
end. So eventually there are depending on the file size used
some really big buffers, or something else.

rolf


----------------------------------------------------------------------

Comment By: Karl Waclawek (kwaclaw)
Date: 2002-11-20 11:36

Message:
Logged In: YES 
user_id=290026

I fixed a small memory leak, but don't think this
applies here - the leak would only occur under certain
error conditions.

I couldn't really find a problem in the xmlwf code
at a first check. Does this memory leak also happen 
with older versions of  the expat lib?

----------------------------------------------------------------------

Comment By: Karl Waclawek (kwaclaw)
Date: 2002-11-20 11:07

Message:
Logged In: YES 
user_id=290026

What options are you using when calling xmlwf?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=110127&aid=632146&group_id=10127



More information about the Expat-bugs mailing list