[XML-SIG] XML parser benchmark

Robb Shecter shecter@darmstadt.gmd.de
Mon, 10 May 1999 10:28:57 +0200


Lars Marius Garshol wrote:

> This was posted by Clark Cooper to xml-dev yesterday. ...

> |
> | That compares the performance of 6 XML parsers using 4 languages:
> |

Hmm... I read the article and checked out the perl script: It looks to me like
there's a serious problem with how the test was conducted.  Maybe I don't
understand what's going on, but this looks obvious:

The tests were apparently done with the unix "time" command, by shelling out, and
starting a new process for each document. This means that the interpreter-based
languages get hit with two disadvantages: 1) They're penalized for VM startup and
shutdown times.  2) After parsing a document, all loaded objects, references,
cached whatevers and knowedge gained are thrown away, and can't be used for the
next document.

To me, this is a valid issue because the test environment is artificial: it
doesn't represent real-world uses.  It also isn't the execution style for which
the systems (IMO) were designed.  The test most closely models a CGI environment,
which is a dying programming style.

My guess is that if the test more closely modelled real-world use: a server that,
in its lifetime, parses many documents, then the results may not have been so
exagerated.

- Robb