[Expat-bugs] [ expat-Bugs-569461 ] OASIS XML Test Suite

Wed Sep 24 19:48:15 EDT 2003

Bugs item #569461, was opened at 2002-06-15 15:34
Message generated for change (Comment added) made by kwaclaw
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=110127&aid=569461&group_id=10127

Category: None
Group: None
Status: Open
Resolution: Accepted
Priority: 5
Submitted By: Rolf Ade (pointsman)
Assigned to: Karl Waclawek (kwaclaw)
>Summary: OASIS XML Test Suite

Initial Comment:
I've tested expat-1.95.3 (with xmltok.c updated to rev.

1.17, becase of bug 566240, all other files are the

original 1.95.3) against the recently updated OASIS xml

test suite (XML 1.0 (Second Edition) errata 20020320,

W3C Conformance Test Suite 20020606), avaliable via

http://www.w3.org/XML/Test/ and found a few new

problems, that are not triggered by older versions of

this test suite.

As in previous reports, I checked all

not-wellformedness tests (should

all raise error) and all valid tests (should all pass)

of the

test-suites xmltest, ibm, sun and oasis with xmlwf -p.

Especially for the well-formedness tests, I have _not_

throughout checked if the error reason, reported by

expat is the expected error, but checked only

mechanical, if the test has raised an error, regardless

of the exact error reason. This method is clearly not

perfect, and this time we have an example, that

underlines this.

ibm/not-wf/P32/ibm32n09.xml

This is a new test, not included in previous versions.

Problem is, that the standalone document declaration

has the value "yes" and there is an external markup

declaration of an entity (other than amp, lt, gt, apos,

quot). xmlwf -p doesn't report an error. The not

well-formedness problem is, that standalone="yes"

means, that all informations needed to build the XML

infoset must be found in the document entity

(standalone="yes" doesn't mean, that the document must

not have an external subset or external PE's, only that

this external entities doesn't change - per attribute

defaults or as in this case, entity declarations -

change the info in the document entity. See the last

sentence of "Well-Formedness Constraint: Entity

Declared" (P68).

ibm/not-wf/P68/ibm68n06.xml

Same reason as the test befor. This test _was_ present

in previous versions of the test suite. But with the

previous version of the external subset of this test,

xmlwf claimed a "syntax error" error in the external

subset, which I plain can't understand (eventually an

other expat bug?), but is clearly not the expected

error. In the new version of the test suite, this

external subset now has an XML declaration with

explicite encoding (the older version had only an XML

declaration without encoding) and is accepted by expat.

xmltest/not-wf/not-sa/010.xml

xmltest/not-wf/not-sa/011.xml

This tests are new in this edition of the test suite.

Unfortunately, this both tests seems to be not

documented, either in the test files isself nor in the

documentation file xmlconf-20020606.htm. As far as I

see, this tests test "Validity Constraint: Proper

Declaration/PE Nesting" (P29).

xmltest/not-wf/not-sa/005.xml

This test raised error with previous expat versions,

but does not anymore due to the changes, discussed in

bug 548690. This is intentional, according to the

548690 discussion. This test is now listed under "XML

Documents with Optional Errors". The test suite

documentation says:

"Conforming XML 1.0 Processors are permitted to ignore

certain errors, or to report them at user option. In

this section of this test report are found descriptions

of test cases which fit into this category.

Processor behavior on such test cases does not affect

conformance to the XML 1.0 (Second Edition)

Recommendation, except as noted." 

So, according to this, it's OK, that expat doesn't

report an error for this case. Since both reporting

error and not reporting error are OK, it may be

debatably, which behavior is more convenient for the

expat user. (Karl: ;-))

sun/not-wf/not-sa03.xml

This is a new test in this edition of the test suite.

Unfortunately, this test seems not to be documented. As

far as I see, it tests the same as

xmltest/not-wf/not-sa/005.xml

Tests, that still are wrong, as in previous versions are

ibm/not-wf/misc/432gewf.xml

sun/not-wf/uri01.xml

These are already discussed in the past.

Well, that's all.

rolf

----------------------------------------------------------------------

>Comment By: Karl Waclawek (kwaclaw)
Date: 2003-09-24 19:48

Message:
Logged In: YES 
user_id=290026

Just changed the summary to be more generic, as this bug 

will probably stay open permanently, assuming we will never

pass 100% of all test cases.

----------------------------------------------------------------------

Comment By: Karl Waclawek (kwaclaw)
Date: 2003-01-29 16:26

Message:
Logged In: YES 
user_id=290026

This is just to report that the new release Expat 1.95.6

passes the OASIS test suite (same version - 20020606)

with the exact same results as Expat 1.95.5.

----------------------------------------------------------------------

Comment By: Karl Waclawek (kwaclaw)
Date: 2003-01-21 14:21

Message:
Logged In: YES 
user_id=290026

For comparison purposes, I ran the xmltest.sh test script

against release 1.95.5. The results are attached as

TestResults_1_95_5.txt.

Discussion of results:

There are many cases were "output differs"is reported,

but these are due to xmlwf having a different definition

of "canonical XML" than used in the test suite.

Leaving these out, and those that were discussed 

already in this thread, we have the following errors 

reported:

(Note: the two test cases 

          ibm/not-wf/P32/ibm32n09.xml and

          ibm/not-wf/P68/ibm68n06.xml

          are not reported anymore by the script)

* In ibm/invalid/P49/: ibm49i02.xml:7:1: error in 

   processing external entity reference:

   The associated DTD file does not exist - an error

   in the test suite.

The next three documents are not UTF-8 encoded,

and do not have an XML declaration, so Expat rejects 

them, which is correct. An error in the test suite.

* In xmltest/valid/sa/: 049.xml:2:0: not well-formed 

   (invalid token)

* In xmltest/valid/sa/: 050.xml:2:0: not well-formed 

   (invalid token)

* In xmltest/valid/sa/: 051.xml:2:0: not well-formed 

   (invalid token)

The next two documents are classified as invalid,

but well-formed, but they contain faulty UTF-16 

encoding, so they should be classified as not

well-formed. Expat seems correct here.

* In sun/invalid/: utf16b.xml:2:0: not well-formed 

   (invalid token)

* In sun/invalid/: utf16l.xml:1:40: not well-formed 

   (invalid token)

The next three are not marked as why they should fail, 

so the script thinks they are not well-formed, but in

fact they are:

* Well formed: oasis/p06fail1.xml

* Well formed: oasis/p08fail1.xml

* Well formed: oasis/p08fail2.xml

So, no new test case errors have really been added

for release 1.95.5.

----------------------------------------------------------------------

Comment By: Karl Waclawek (kwaclaw)
Date: 2003-01-19 14:01

Message:
Logged In: YES 
user_id=290026

Just a comment:

This bug report will likely stay open until Expat passes the

OASIS test suite without any problem at all.

Since no parser currently achieves this, there is a good

chance this bug report will stay open for a long time to come..

----------------------------------------------------------------------

Comment By: Karl Waclawek (kwaclaw)
Date: 2002-07-07 11:39

Message:
Logged In: YES 
user_id=290026

Patch #587161 should fix some of the problems,

but I specifically made no attempt to fix the problems

Expat has with:

- xmltest/not-wf/not-sa/010.xml and

- xmltest/not-wf/not-sa/011.xml.

Reason: It turns out, after consulting with the mailing

list for the XML test suite, public-xml-testsuite at w3.org,

that these two violate WFC: PE Between Declarations.

There is no quick and easy fix for this in Expat,

and I would have to spend some time thinking about it,

which I don't have at the moment.

Karl

----------------------------------------------------------------------

Comment By: Karl Waclawek (kwaclaw)
Date: 2002-06-17 23:15

Message:
Logged In: YES 
user_id=290026

Assigned to me, but only for the three test cases

described in my last message.

Karl

----------------------------------------------------------------------

Comment By: Rolf Ade (pointsman)
Date: 2002-06-17 19:21

Message:
Logged In: YES 
user_id=13222

Agreed

----------------------------------------------------------------------

Comment By: Karl Waclawek (kwaclaw)
Date: 2002-06-17 15:05

Message:
Logged In: YES 
user_id=290026

Given an improved understanding of section 4.1

in the XML spec, I will try to fix the following

test cases in the next Expat release:

ibm/not-wf/P32/ibm32n09.xml,

ibm/not-wf/P68/ibm68n06.xml and

sun/not-wf/not-sa03.xml 

In my opinion, the third one is not the same type as

xmltest/not-wf/not-sa/005.xml, but the same type

as the other two.

About the test cases

xmltest/not-wf/not-sa/010.xml and

xmltest/not-wf/not-sa/011.xml:

If they really check validity constraint P29, as Rolf

has suggested, then it is OK that Expat does not report

an error.

So, If I am successful, we would be left with only:

ibm/not-wf/misc/432gewf.xml and

sun/not-wf/uri01.xml,

conformance with which does not seem a 100% 

necessity, as previously discussed.

Karl

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=110127&aid=569461&group_id=10127