Mailman 3 August 2008 - lxml - The Python XML Toolkit

[lxml-dev] lxml built for python2.3 on windows
by Nguyễn Khánh Duy Aug. 22, 2008

Aug. 22, 2008

Does anyone have the binaries for lxml on windows for python 2.3 ? I could not build it and cannot find the prebuilt anywhere. -- Nguyen Khanh Duy nguyenkhanhduy(a)gmail.com

1 0

[lxml-dev] Undefined preference when building lxml xsltLibxsltVersion
by Nguyễn Khánh Duy Aug. 21, 2008

Aug. 21, 2008

I am trying to build lxml for python 2.3 on windows, using mingw32. I've successfully downloaded and installed libxslt and libxml2, put the include header files and the lib files where they need to be, but when I build lxml the following error appears: Am I missing any .lib files? libxslt, libxml2, libexslt, iconv and zlib are all in the python 2.3 libs folder. D:\LXML\lxml-2.1.1>python setup.py build -cmingw32 Building lxml version 2.1.1. NOTE: Trying to build without Cython, pre-generated 'src/lxml/lxml.etree.c' needs to be available. ERROR: 'xslt-config' is not recognized as an internal or external command, operable program or batch file. ** make sure the development packages of libxml2 and libxslt are installed ** Using build configuration of libxslt running build running build_py running build_ext building 'lxml.etree' extension writing build\temp.win32-2.3\Release\src\lxml\etree.def C:\MinGW\bin\gcc.exe -mno-cygwin -shared -s build\temp.win32-2.3\Release\src\lxm l\lxml.etree.o build\temp.win32-2.3\Release\src\lxml\etree.def -LD:\Python23\lib s -LD:\Python23\PCBuild -llibxslt -llibexslt -llibxml2 -liconv -lzlib -lWS2_32 - lpython23 -o build\lib.win32-2.3\lxml\etree.pyd build\temp.win32-2.3\Release\src\lxml\lxml.etree.o:lxml.etree.c:(.text+0x67e20): undefined reference to `xsltProcessOneNode' build\temp.win32-2.3\Release\src\lxml\lxml.etree.o:lxml.etree.c:(.text+0x83946): undefined reference to `xsltLibxsltVersion' build\temp.win32-2.3\Release\src\lxml\lxml.etree.o:lxml.etree.c:(.text+0x839e2): undefined reference to `xsltDocDefaultLoader' collect2: ld returned 1 exit status error: command 'gcc' failed with exit status 1 -- Nguyen Khanh Duy nguyenkhanhduy(a)gmail.com

2 3

[lxml-dev] objectify.ObjectPath("wrongroot.a.b")(root, None) should not raise an exception
by Holger Joukl Aug. 21, 2008

Aug. 21, 2008

Hi, using an absolute ObjectPath with a wrong root element objectify.ObjectPath("wrongroot.a.b")(root, None) currently raises ValueError even if a default is given: >>> root = objectify.fromstring("<root><a><b>23</b></a></root>") >>> objectify.ObjectPath("root.a.b")(root, None) 23 >>> objectify.ObjectPath("wrongroot.a.b")(root, None) Traceback (most recent call last): File "<stdin>", line 1, in ? File "objectpath.pxi", line 53, in lxml.objectify.ObjectPath.__call__ File "objectpath.pxi", line 197, in lxml.objectify._findObjectPath ValueError: root element does not match: need wrongroot, got root >>> Whereas ObjectPath returns a given default if any other element of the path is not in the tree: >>> objectify.ObjectPath("root.wronga.b")(root, None) >>> I'll change this in trunk to gracefully return the default, and add some tests (unless someone stops me... :) Holger -- Psssst! Schon das coole Video vom GMX MultiMessenger gesehen? Der Eine für Alle: http://www.gmx.net/de/go/messenger03

3 2

[lxml-dev] Fwd: [xml] Security fix for libxml2
by Stefan Behnel Aug. 20, 2008

Aug. 20, 2008

FYI -------- Original-Message -------- Subject: [xml] Security fix for libxml2 Date: Wed, 20 Aug 2008 19:00:51 +0200 From: Daniel Veillard <veillard(a)redhat.com> To: xml(a)gnome.org Bad news, when checking against recursive entities expansion problem back when it was made official (c.f. the billion laught attack circa 2004) I had checked for the normal recursion, but when happening in an attribute value the resource consumption is way faster and the recursion detection in place is not sufficient to catch the problem. Basically when this happen within an attribute just checking for a recursion depth is not sufficient, and the only good method I could find was to count the number of entities replacement taking place while parsing a given document, and drop parsing after half a million substitution. I think it's a fair default process and what the patches below implements for various libxml2 versions, but i can understand that in some case that may be problematic. So i intend in the next release (2.7.0 hopefully available soon) to add a parser flag removing the hardcoded limits (there is also a maximum document depth in place). Distributions have been made aware of the problem for a couple of weeks and updates should be available soon from normal update channels I'm updating SVN with the fix too, Daniel Index: include/libxml/parser.h =================================================================== --- include/libxml/parser.h (revision 3771) +++ include/libxml/parser.h (working copy) @@ -297,6 +297,7 @@ struct _xmlParserCtxt { */ xmlError lastError; xmlParserMode parseMode; /* the parser mode */ + unsigned long nbentities; /* number of entities references */ }; /** Index: include/libxml/entities.h =================================================================== --- include/libxml/entities.h (revision 3771) +++ include/libxml/entities.h (working copy) @@ -57,6 +57,7 @@ struct _xmlEntity { const xmlChar *URI; /* the full URI as computed */ int owner; /* does the entity own the childrens */ int checked; /* was the entity content checked */ + unsigned long nbentities; /* the number of entities references */ }; /* Index: entities.c =================================================================== --- entities.c (revision 3771) +++ entities.c (working copy) @@ -31,35 +31,35 @@ static xmlEntity xmlEntityLt = { NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST "<", BAD_CAST "<", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0, 1 + NULL, NULL, NULL, NULL, 0, 1, 0 }; static xmlEntity xmlEntityGt = { NULL, XML_ENTITY_DECL, BAD_CAST "gt", NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST ">", BAD_CAST ">", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0, 1 + NULL, NULL, NULL, NULL, 0, 1, 0 }; static xmlEntity xmlEntityAmp = { NULL, XML_ENTITY_DECL, BAD_CAST "amp", NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST "&", BAD_CAST "&", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0, 1 + NULL, NULL, NULL, NULL, 0, 1, 0 }; static xmlEntity xmlEntityQuot = { NULL, XML_ENTITY_DECL, BAD_CAST "quot", NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST "\"", BAD_CAST "\"", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0, 1 + NULL, NULL, NULL, NULL, 0, 1, 0 }; static xmlEntity xmlEntityApos = { NULL, XML_ENTITY_DECL, BAD_CAST "apos", NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST "'", BAD_CAST "'", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0, 1 + NULL, NULL, NULL, NULL, 0, 1, 0 }; /** Index: parserInternals.c =================================================================== --- parserInternals.c (revision 3771) +++ parserInternals.c (working copy) @@ -1670,6 +1670,7 @@ xmlInitParserCtxt(xmlParserCtxtPtr ctxt) ctxt->depth = 0; ctxt->charset = XML_CHAR_ENCODING_UTF8; ctxt->catalogs = NULL; + ctxt->nbentities = 0; xmlInitNodeInfoSeq(&ctxt->node_seq); return(0); } Index: parser.c =================================================================== --- parser.c (revision 3771) +++ parser.c (working copy) @@ -2379,7 +2379,7 @@ xmlStringLenDecodeEntities(xmlParserCtxt return(NULL); last = str + len; - if (ctxt->depth > 40) { + if ((ctxt->depth > 40) || (ctxt->nbentities >= 500000)) { xmlFatalErr(ctxt, XML_ERR_ENTITY_LOOP, NULL); return(NULL); } @@ -2417,6 +2417,11 @@ xmlStringLenDecodeEntities(xmlParserCtxt "String decoding Entity Reference: %.30s\n", str); ent = xmlParseStringEntityRef(ctxt, &str); + if (ctxt->lastError.code == XML_ERR_ENTITY_LOOP) + goto int_error; + ctxt->nbentities++; + if (ent != NULL) + ctxt->nbentities += ent->nbentities; if ((ent != NULL) && (ent->etype == XML_INTERNAL_PREDEFINED_ENTITY)) { if (ent->content != NULL) { @@ -2462,6 +2467,11 @@ xmlStringLenDecodeEntities(xmlParserCtxt xmlGenericError(xmlGenericErrorContext, "String decoding PE Reference: %.30s\n", str); ent = xmlParseStringPEReference(ctxt, &str); + if (ctxt->lastError.code == XML_ERR_ENTITY_LOOP) + goto int_error; + ctxt->nbentities++; + if (ent != NULL) + ctxt->nbentities += ent->nbentities; if (ent != NULL) { if (ent->content == NULL) { if (xmlLoadEntityContent(ctxt, ent) < 0) { @@ -2501,6 +2511,7 @@ xmlStringLenDecodeEntities(xmlParserCtxt mem_error: xmlErrMemory(ctxt, NULL); +int_error: if (rep != NULL) xmlFree(rep); if (buffer != NULL) @@ -3542,6 +3553,9 @@ xmlParseAttValueComplex(xmlParserCtxtPtr } } else { ent = xmlParseEntityRef(ctxt); + ctxt->nbentities++; + if (ent != NULL) + ctxt->nbentities += ent->nbentities; if ((ent != NULL) && (ent->etype == XML_INTERNAL_PREDEFINED_ENTITY)) { if (len > buf_size - 10) { @@ -4844,6 +4858,7 @@ xmlParseEntityDecl(xmlParserCtxtPtr ctxt int isParameter = 0; xmlChar *orig = NULL; int skipped; + unsigned long oldnbent = ctxt->nbentities; /* GROW; done in the caller */ if (CMP8(CUR_PTR, '<', '!', 'E', 'N', 'T', 'I', 'T', 'Y')) { @@ -5068,6 +5083,7 @@ xmlParseEntityDecl(xmlParserCtxtPtr ctxt } } if (cur != NULL) { + cur->nbentities = ctxt->nbentities - oldnbent; if (cur->orig != NULL) xmlFree(orig); else @@ -6477,6 +6493,11 @@ xmlParseReference(xmlParserCtxtPtr ctxt) if (ent == NULL) return; if (!ctxt->wellFormed) return; + ctxt->nbentities++; + if (ctxt->nbentities >= 500000) { + xmlFatalErr(ctxt, XML_ERR_ENTITY_LOOP, NULL); + return; + } was_checked = ent->checked; if ((ent->name != NULL) && (ent->etype != XML_INTERNAL_PREDEFINED_ENTITY)) { @@ -6537,6 +6558,7 @@ xmlParseReference(xmlParserCtxtPtr ctxt) xmlFreeNodeList(list); } } else { + unsigned long oldnbent = ctxt->nbentities; /* * 4.3.2: An internal general parsed entity is well-formed * if its replacement text matches the production labeled @@ -6559,6 +6581,7 @@ xmlParseReference(xmlParserCtxtPtr ctxt) ret = xmlParseBalancedChunkMemoryInternal(ctxt, value, user_data, &list); ctxt->depth--; + } else if (ent->etype == XML_EXTERNAL_GENERAL_PARSED_ENTITY) { ctxt->depth++; @@ -6571,6 +6594,7 @@ xmlParseReference(xmlParserCtxtPtr ctxt) xmlErrMsgStr(ctxt, XML_ERR_INTERNAL_ERROR, "invalid entity type found\n", NULL); } + ent->nbentities = ctxt->nbentities - oldnbent; if (ret == XML_ERR_ENTITY_LOOP) { xmlFatalErr(ctxt, XML_ERR_ENTITY_LOOP, NULL); return; @@ -6629,6 +6653,7 @@ xmlParseReference(xmlParserCtxtPtr ctxt) } ent->checked = 1; } + ctxt->nbentities += ent->nbentities; if (ent->children == NULL) { /* @@ -11800,7 +11825,7 @@ xmlParseCtxtExternalEntity(xmlParserCtxt if (ctx == NULL) return(-1); - if (ctx->depth > 40) { + if ((ctx->depth > 40) || (ctx->nbentities >= 500000)) { return(XML_ERR_ENTITY_LOOP); } @@ -12010,7 +12035,8 @@ xmlParseExternalEntityPrivate(xmlDocPtr xmlChar start[4]; xmlCharEncoding enc; - if (depth > 40) { + if ((depth > 40) || + ((oldctxt != NULL) && (oldctxt->nbentities >= 500000))) { return(XML_ERR_ENTITY_LOOP); } @@ -12154,6 +12180,7 @@ xmlParseExternalEntityPrivate(xmlDocPtr oldctxt->node_seq.maximum = ctxt->node_seq.maximum; oldctxt->node_seq.length = ctxt->node_seq.length; oldctxt->node_seq.buffer = ctxt->node_seq.buffer; + oldctxt->nbentities += ctxt->nbentities; ctxt->node_seq.maximum = 0; ctxt->node_seq.length = 0; ctxt->node_seq.buffer = NULL; @@ -12254,7 +12281,7 @@ xmlParseBalancedChunkMemoryInternal(xmlP int size; xmlParserErrors ret = XML_ERR_OK; - if (oldctxt->depth > 40) { + if ((oldctxt->depth > 40) || (oldctxt->nbentities >= 500000)) { return(XML_ERR_ENTITY_LOOP); } @@ -12379,6 +12406,7 @@ xmlParseBalancedChunkMemoryInternal(xmlP ctxt->myDoc->last = last; } + oldctxt->nbentities += ctxt->nbentities; ctxt->sax = oldsax; ctxt->dict = NULL; ctxt->attsDefault = NULL; @@ -13695,6 +13723,7 @@ xmlCtxtReset(xmlParserCtxtPtr ctxt) ctxt->depth = 0; ctxt->charset = XML_CHAR_ENCODING_UTF8; ctxt->catalogs = NULL; + ctxt->nbentities = 0; xmlInitNodeInfoSeq(&ctxt->node_seq); if (ctxt->attsDefault != NULL) { --- include/libxml/parser.h (revision 3771) +++ include/libxml/parser.h (working copy) @@ -297,6 +297,7 @@ struct _xmlParserCtxt { */ xmlError lastError; xmlParserMode parseMode; /* the parser mode */ + unsigned long nbentities; /* number of entities references */ }; /** --- include/libxml/entities.h.orig 2005-01-04 15:49:49.000000000 +0100 +++ include/libxml/entities.h 2008-08-11 17:56:53.000000000 +0200 @@ -56,6 +56,7 @@ struct _xmlEntity { struct _xmlEntity *nexte; /* unused */ const xmlChar *URI; /* the full URI as computed */ int owner; /* does the entity own the childrens */ + unsigned long nbentities; /* the number of entities references */ }; /* --- entities.c.orig 2006-03-09 17:39:46.000000000 +0100 +++ entities.c 2008-08-11 18:01:06.000000000 +0200 @@ -31,35 +31,35 @@ static xmlEntity xmlEntityLt = { NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST "<", BAD_CAST "<", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0 + NULL, NULL, NULL, NULL, 0, 0 }; static xmlEntity xmlEntityGt = { NULL, XML_ENTITY_DECL, BAD_CAST "gt", NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST ">", BAD_CAST ">", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0 + NULL, NULL, NULL, NULL, 0, 0 }; static xmlEntity xmlEntityAmp = { NULL, XML_ENTITY_DECL, BAD_CAST "amp", NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST "&", BAD_CAST "&", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0 + NULL, NULL, NULL, NULL, 0, 0 }; static xmlEntity xmlEntityQuot = { NULL, XML_ENTITY_DECL, BAD_CAST "quot", NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST "\"", BAD_CAST "\"", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0 + NULL, NULL, NULL, NULL, 0, 0 }; static xmlEntity xmlEntityApos = { NULL, XML_ENTITY_DECL, BAD_CAST "apos", NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST "'", BAD_CAST "'", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0 + NULL, NULL, NULL, NULL, 0, 0 }; /** --- parser.c.orig 2006-04-23 11:39:15.000000000 +0200 +++ parser.c 2008-08-11 18:36:56.000000000 +0200 @@ -2174,7 +2176,7 @@ xmlStringLenDecodeEntities(xmlParserCtxt return(NULL); last = str + len; - if (ctxt->depth > 40) { + if ((ctxt->depth > 40) || (ctxt->nbentities >= 500000)) { xmlFatalErr(ctxt, XML_ERR_ENTITY_LOOP, NULL); return(NULL); } @@ -2212,6 +2214,11 @@ xmlStringLenDecodeEntities(xmlParserCtxt "String decoding Entity Reference: %.30s\n", str); ent = xmlParseStringEntityRef(ctxt, &str); + if (ctxt->lastError.code == XML_ERR_ENTITY_LOOP) + goto int_error; + ctxt->nbentities++; + if (ent != NULL) + ctxt->nbentities += ent->nbentities; if ((ent != NULL) && (ent->etype == XML_INTERNAL_PREDEFINED_ENTITY)) { if (ent->content != NULL) { @@ -2258,6 +2265,11 @@ xmlStringLenDecodeEntities(xmlParserCtxt xmlGenericError(xmlGenericErrorContext, "String decoding PE Reference: %.30s\n", str); ent = xmlParseStringPEReference(ctxt, &str); + if (ctxt->lastError.code == XML_ERR_ENTITY_LOOP) + goto int_error; + ctxt->nbentities++; + if (ent != NULL) + ctxt->nbentities += ent->nbentities; if (ent != NULL) { xmlChar *rep; @@ -2294,6 +2306,9 @@ xmlStringLenDecodeEntities(xmlParserCtxt mem_error: xmlErrMemory(ctxt, NULL); +int_error: + if (buffer != NULL) + xmlFree(buffer); return(NULL); } @@ -3100,6 +3115,9 @@ xmlParseAttValueComplex(xmlParserCtxtPtr } } else { ent = xmlParseEntityRef(ctxt); + ctxt->nbentities++; + if (ent != NULL) + ctxt->nbentities += ent->nbentities; if ((ent != NULL) && (ent->etype == XML_INTERNAL_PREDEFINED_ENTITY)) { if (len > buf_size - 10) { @@ -4342,6 +4360,7 @@ xmlParseEntityDecl(xmlParserCtxtPtr ctxt int isParameter = 0; xmlChar *orig = NULL; int skipped; + unsigned long oldnbent = ctxt->nbentities; /* GROW; done in the caller */ if (CMP8(CUR_PTR, '<', '!', 'E', 'N', 'T', 'I', 'T', 'Y')) { @@ -4551,6 +4570,7 @@ xmlParseEntityDecl(xmlParserCtxtPtr ctxt } } if (cur != NULL) { + cur->nbentities = ctxt->nbentities - oldnbent; if (cur->orig != NULL) xmlFree(orig); else @@ -5927,6 +5947,11 @@ xmlParseReference(xmlParserCtxtPtr ctxt) if (ent == NULL) return; if (!ctxt->wellFormed) return; + ctxt->nbentities++; + if (ctxt->nbentities >= 500000) { + xmlFatalErr(ctxt, XML_ERR_ENTITY_LOOP, NULL); + return; + } if ((ent->name != NULL) && (ent->etype != XML_INTERNAL_PREDEFINED_ENTITY)) { xmlNodePtr list = NULL; @@ -5985,6 +6010,7 @@ xmlParseReference(xmlParserCtxtPtr ctxt) xmlFreeNodeList(list); } } else { + unsigned long oldnbent = ctxt->nbentities; /* * 4.3.2: An internal general parsed entity is well-formed * if its replacement text matches the production labeled @@ -6007,6 +6033,7 @@ xmlParseReference(xmlParserCtxtPtr ctxt) ret = xmlParseBalancedChunkMemoryInternal(ctxt, value, user_data, &list); ctxt->depth--; + } else if (ent->etype == XML_EXTERNAL_GENERAL_PARSED_ENTITY) { ctxt->depth++; @@ -6019,6 +6046,7 @@ xmlParseReference(xmlParserCtxtPtr ctxt) xmlErrMsgStr(ctxt, XML_ERR_INTERNAL_ERROR, "invalid entity type found\n", NULL); } + ent->nbentities = ctxt->nbentities - oldnbent; if (ret == XML_ERR_ENTITY_LOOP) { xmlFatalErr(ctxt, XML_ERR_ENTITY_LOOP, NULL); return; @@ -6075,6 +6103,7 @@ xmlParseReference(xmlParserCtxtPtr ctxt) } } } + ctxt->nbentities += ent->nbentities; if ((ctxt->sax != NULL) && (ctxt->sax->reference != NULL) && (ctxt->replaceEntities == 0) && (!ctxt->disableSAX)) { /* @@ -11035,7 +11064,7 @@ xmlParseCtxtExternalEntity(xmlParserCtxt if (ctx == NULL) return(-1); - if (ctx->depth > 40) { + if ((ctx->depth > 40) || (ctx->nbentities >= 500000)) { return(XML_ERR_ENTITY_LOOP); } @@ -11220,7 +11249,8 @@ xmlParseExternalEntityPrivate(xmlDocPtr xmlChar start[4]; xmlCharEncoding enc; - if (depth > 40) { + if ((depth > 40) || + ((oldctxt != NULL) && (oldctxt->nbentities >= 500000))) { return(XML_ERR_ENTITY_LOOP); } @@ -11363,6 +11393,7 @@ xmlParseExternalEntityPrivate(xmlDocPtr oldctxt->node_seq.maximum = ctxt->node_seq.maximum; oldctxt->node_seq.length = ctxt->node_seq.length; oldctxt->node_seq.buffer = ctxt->node_seq.buffer; + oldctxt->nbentities += ctxt->nbentities; ctxt->node_seq.maximum = 0; ctxt->node_seq.length = 0; ctxt->node_seq.buffer = NULL; @@ -11463,7 +11494,7 @@ xmlParseBalancedChunkMemoryInternal(xmlP int size; xmlParserErrors ret = XML_ERR_OK; - if (oldctxt->depth > 40) { + if ((oldctxt->depth > 40) || (oldctxt->nbentities >= 500000)) { return(XML_ERR_ENTITY_LOOP); } @@ -11587,6 +11618,7 @@ xmlParseBalancedChunkMemoryInternal(xmlP ctxt->myDoc->last = last; } + oldctxt->nbentities += ctxt->nbentities; ctxt->sax = oldsax; ctxt->dict = NULL; ctxt->attsDefault = NULL; @@ -12883,6 +12915,7 @@ xmlCtxtReset(xmlParserCtxtPtr ctxt) ctxt->depth = 0; ctxt->charset = XML_CHAR_ENCODING_UTF8; ctxt->catalogs = NULL; + ctxt->nbentities = 0; xmlInitNodeInfoSeq(&ctxt->node_seq); if (ctxt->attsDefault != NULL) { --- include/libxml/parser.h (revision 3771) +++ include/libxml/parser.h (working copy) @@ -297,6 +297,7 @@ struct _xmlParserCtxt { */ xmlError lastError; xmlParserMode parseMode; /* the parser mode */ + unsigned long nbentities; /* number of entities references */ }; /** --- include/libxml/entities.h.orig 2005-01-04 15:49:49.000000000 +0100 +++ include/libxml/entities.h 2008-08-11 17:56:53.000000000 +0200 @@ -56,6 +56,7 @@ struct _xmlEntity { struct _xmlEntity *nexte; /* unused */ const xmlChar *URI; /* the full URI as computed */ int owner; /* does the entity own the childrens */ + unsigned long nbentities; /* the number of entities references */ }; /* --- entities.c.orig 2006-03-09 17:39:46.000000000 +0100 +++ entities.c 2008-08-11 18:01:06.000000000 +0200 @@ -31,35 +31,35 @@ static xmlEntity xmlEntityLt = { NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST "<", BAD_CAST "<", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0 + NULL, NULL, NULL, NULL, 0, 0 }; static xmlEntity xmlEntityGt = { NULL, XML_ENTITY_DECL, BAD_CAST "gt", NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST ">", BAD_CAST ">", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0 + NULL, NULL, NULL, NULL, 0, 0 }; static xmlEntity xmlEntityAmp = { NULL, XML_ENTITY_DECL, BAD_CAST "amp", NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST "&", BAD_CAST "&", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0 + NULL, NULL, NULL, NULL, 0, 0 }; static xmlEntity xmlEntityQuot = { NULL, XML_ENTITY_DECL, BAD_CAST "quot", NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST "\"", BAD_CAST "\"", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0 + NULL, NULL, NULL, NULL, 0, 0 }; static xmlEntity xmlEntityApos = { NULL, XML_ENTITY_DECL, BAD_CAST "apos", NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST "'", BAD_CAST "'", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0 + NULL, NULL, NULL, NULL, 0, 0 }; /** --- parser.c.orig 2006-04-23 11:39:15.000000000 +0200 +++ parser.c 2008-08-11 18:36:56.000000000 +0200 @@ -2174,7 +2176,7 @@ xmlStringLenDecodeEntities(xmlParserCtxt return(NULL); last = str + len; - if (ctxt->depth > 40) { + if ((ctxt->depth > 40) || (ctxt->nbentities >= 500000)) { xmlFatalErr(ctxt, XML_ERR_ENTITY_LOOP, NULL); return(NULL); } @@ -2212,6 +2214,11 @@ xmlStringLenDecodeEntities(xmlParserCtxt "String decoding Entity Reference: %.30s\n", str); ent = xmlParseStringEntityRef(ctxt, &str); + if (ctxt->lastError.code == XML_ERR_ENTITY_LOOP) + goto int_error; + ctxt->nbentities++; + if (ent != NULL) + ctxt->nbentities += ent->nbentities; if ((ent != NULL) && (ent->etype == XML_INTERNAL_PREDEFINED_ENTITY)) { if (ent->content != NULL) { @@ -2258,6 +2265,11 @@ xmlStringLenDecodeEntities(xmlParserCtxt xmlGenericError(xmlGenericErrorContext, "String decoding PE Reference: %.30s\n", str); ent = xmlParseStringPEReference(ctxt, &str); + if (ctxt->lastError.code == XML_ERR_ENTITY_LOOP) + goto int_error; + ctxt->nbentities++; + if (ent != NULL) + ctxt->nbentities += ent->nbentities; if (ent != NULL) { xmlChar *rep; @@ -2294,6 +2306,9 @@ xmlStringLenDecodeEntities(xmlParserCtxt mem_error: xmlErrMemory(ctxt, NULL); +int_error: + if (buffer != NULL) + xmlFree(buffer); return(NULL); } @@ -3100,6 +3115,9 @@ xmlParseAttValueComplex(xmlParserCtxtPtr } } else { ent = xmlParseEntityRef(ctxt); + ctxt->nbentities++; + if (ent != NULL) + ctxt->nbentities += ent->nbentities; if ((ent != NULL) && (ent->etype == XML_INTERNAL_PREDEFINED_ENTITY)) { if (len > buf_size - 10) { @@ -4342,6 +4360,7 @@ xmlParseEntityDecl(xmlParserCtxtPtr ctxt int isParameter = 0; xmlChar *orig = NULL; int skipped; + unsigned long oldnbent = ctxt->nbentities; /* GROW; done in the caller */ if (CMP8(CUR_PTR, '<', '!', 'E', 'N', 'T', 'I', 'T', 'Y')) { @@ -4551,6 +4570,7 @@ xmlParseEntityDecl(xmlParserCtxtPtr ctxt } } if (cur != NULL) { + cur->nbentities = ctxt->nbentities - oldnbent; if (cur->orig != NULL) xmlFree(orig); else @@ -5927,6 +5947,11 @@ xmlParseReference(xmlParserCtxtPtr ctxt) if (ent == NULL) return; if (!ctxt->wellFormed) return; + ctxt->nbentities++; + if (ctxt->nbentities >= 500000) { + xmlFatalErr(ctxt, XML_ERR_ENTITY_LOOP, NULL); + return; + } if ((ent->name != NULL) && (ent->etype != XML_INTERNAL_PREDEFINED_ENTITY)) { xmlNodePtr list = NULL; @@ -5985,6 +6010,7 @@ xmlParseReference(xmlParserCtxtPtr ctxt) xmlFreeNodeList(list); } } else { + unsigned long oldnbent = ctxt->nbentities; /* * 4.3.2: An internal general parsed entity is well-formed * if its replacement text matches the production labeled @@ -6007,6 +6033,7 @@ xmlParseReference(xmlParserCtxtPtr ctxt) ret = xmlParseBalancedChunkMemoryInternal(ctxt, value, user_data, &list); ctxt->depth--; + } else if (ent->etype == XML_EXTERNAL_GENERAL_PARSED_ENTITY) { ctxt->depth++; @@ -6019,6 +6046,7 @@ xmlParseReference(xmlParserCtxtPtr ctxt) xmlErrMsgStr(ctxt, XML_ERR_INTERNAL_ERROR, "invalid entity type found\n", NULL); } + ent->nbentities = ctxt->nbentities - oldnbent; if (ret == XML_ERR_ENTITY_LOOP) { xmlFatalErr(ctxt, XML_ERR_ENTITY_LOOP, NULL); return; @@ -6075,6 +6103,7 @@ xmlParseReference(xmlParserCtxtPtr ctxt) } } } + ctxt->nbentities += ent->nbentities; if ((ctxt->sax != NULL) && (ctxt->sax->reference != NULL) && (ctxt->replaceEntities == 0) && (!ctxt->disableSAX)) { /* @@ -11035,7 +11064,7 @@ xmlParseCtxtExternalEntity(xmlParserCtxt if (ctx == NULL) return(-1); - if (ctx->depth > 40) { + if ((ctx->depth > 40) || (ctx->nbentities >= 500000)) { return(XML_ERR_ENTITY_LOOP); } @@ -11220,7 +11249,8 @@ xmlParseExternalEntityPrivate(xmlDocPtr xmlChar start[4]; xmlCharEncoding enc; - if (depth > 40) { + if ((depth > 40) || + ((oldctxt != NULL) && (oldctxt->nbentities >= 500000))) { return(XML_ERR_ENTITY_LOOP); } @@ -11363,6 +11393,7 @@ xmlParseExternalEntityPrivate(xmlDocPtr oldctxt->node_seq.maximum = ctxt->node_seq.maximum; oldctxt->node_seq.length = ctxt->node_seq.length; oldctxt->node_seq.buffer = ctxt->node_seq.buffer; + oldctxt->nbentities += ctxt->nbentities; ctxt->node_seq.maximum = 0; ctxt->node_seq.length = 0; ctxt->node_seq.buffer = NULL; @@ -11463,7 +11494,7 @@ xmlParseBalancedChunkMemoryInternal(xmlP int size; xmlParserErrors ret = XML_ERR_OK; - if (oldctxt->depth > 40) { + if ((oldctxt->depth > 40) || (oldctxt->nbentities >= 500000)) { return(XML_ERR_ENTITY_LOOP); } @@ -11587,6 +11618,7 @@ xmlParseBalancedChunkMemoryInternal(xmlP ctxt->myDoc->last = last; } + oldctxt->nbentities += ctxt->nbentities; ctxt->sax = oldsax; ctxt->dict = NULL; ctxt->attsDefault = NULL; @@ -12883,6 +12915,7 @@ xmlCtxtReset(xmlParserCtxtPtr ctxt) ctxt->depth = 0; ctxt->charset = XML_CHAR_ENCODING_UTF8; ctxt->catalogs = NULL; + ctxt->nbentities = 0; xmlInitNodeInfoSeq(&ctxt->node_seq); if (ctxt->attsDefault != NULL) {

1 0

[lxml-dev] Failure with custom resolvers and relative xsl:import
by Sidnei da Silva Aug. 19, 2008

Aug. 19, 2008

Hi there, I've got a reproducible failing test when using custom resolvers and relative xsl:import. Basically, if I use resolve_string() or let the default resolver do it's work, everything works fine. If I use resolve_file though, the *next* uri to be resolved will have a relative (to the previous uri resolved) filename, and then there's not enough information available to compute the full URI. This happens with lxml 2.1.1. I am pretty sure it didn't happen with lxml 1.3.x series (which is what I was using before). I'm attaching the problematic test and related files. For reference, this is the problematic code: ... def resolve(self, uri, id, ctx): print uri # return None # works # return self.resolve_string(open(uri, 'r').read(), ctx) # works return self.resolve_file(open(uri, 'r'), ctx) # fails ... -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

4 12

Re: [lxml-dev] Failure with custom resolvers and relative xsl:import
by Stefan Behnel Aug. 19, 2008

Aug. 19, 2008

Mike Meyer wrote: > If your build includes SSL support, then https is in the standard > library. If not, then not. Ah, sure, that's a portability issue then. > Unfortunately, the same is *not* true for server side SSL. ... for which there is tlslite, at least I know that one. From a quick look at their web site, it's supposed to work with httplib also. I never used it either, but a couple of blog entries seem to be happy with it. Stefan

2 1

[lxml-dev] instance from schema
by Peter Aug. 15, 2008

Aug. 15, 2008

I'm looking for a freely available tool that can generate xml instance documents from an xml schema. I'm aware of Sun's xmlgen, but I cannot find a valid web link to it. Does anyone have a current link to Sun's xmlgen or know of a similar tool? When writing such a tool, wouldn't you need to load the schema as an xml document, walk the tree, and output appropriate instance xml (probably not trivial, given xml schema's complexity)? Thank you, Peter

2 1

[lxml-dev] problems with document(''), possibly thread related
by Brad Clements Aug. 15, 2008

Aug. 15, 2008

I have a stylesheet that uses document('') to reference itself. The stylesheet works with xsltproc and xmlstarlet on ubuntu 7.10 However when I use it in a threaded wsgi app with lxml 2.11 or 2.0, it does not work. I then wrote a simple test case (thinking.. aha, I'll report this error), but of course the test case functions correctly. I've spent 4 hours working on this tonight, I'm pooped, and going nuts. basically given an xml document whose root element is "<root />" and a stylesheet that has: <xsl:value-of select="name(document('')/*)" /> From within the threaded wsgi app, the output I get from this is "root", but from the test case and from xsltproc, I get "xsl:stylesheet" My code is more or less like this: ss_parser = etree.XMLParser(load_dtd=True) ss_parser.resolvers.add(Resolver()) stylesheet_doc = etree.fromstring(stylesheet_src, ss_parser, base_url='http://mystylesheet.xsl') stylesheet = etree.XSLT(stylesheet_doc) doc_parser = etree.XMLParser(load_dtd=True) doc_parser.resolvers.add(Resolver()) xml_doc = etree.fromstring(xml_src, doc_parser, base_url='http://myfile.xml') however base_url is some real value when called from wsgi, it's threaded, and my Resolver.resolve method does get called in the wsgi app, but not from the test app. Before I give up, can someone suggest ways in which using lxml from within a threaded app might somehow "break" resolving document(''), but non-threaded it works ok? I don't think I'm using the same parser object for the stylesheet and xml document, the real wsgi code is a tad complicated. However the stylesheet and xml document should be parsed and used within the same thread (which just happens to not be the main thread) I believe this works ok on lxml 1.1.2, but I've already updated my code to use 'base_url' and so forth and I'm too worn out to change all that code just to test a theory. So .. any ideas on what could cause this? thanks for any suggestions.. -- Brad Clements, bkc(a)murkworks.com (315)268-1000 http://www.murkworks.com AOL-IM: BKClements

2 4

Re: [lxml-dev] Segfault on XSLT/XPath undefined variable error.
by Stefan Behnel Aug. 11, 2008

Aug. 11, 2008

Hi, John Krukoff wrote: > First, uninstalled cython and upgraded to 2.1.1, just to make sure this > wasn't already fixed, since there were notes in the changelog about XSLT > logging. With the same method, I can still get the same crash: > > xmlXPathCompiledEval: evaluation failed > Segmentation fault (core dumped) > > With symbols, the backtrace has a bit more information: but still no line numbers ... Could you compile with "-O -ggdb"? > #0 0xb76fb740 in __pyx_f_4lxml_5etree__forwardError () > from /usr/lib/python2.5/site-packages/lxml-2.1.1-py2.5-linux-i686.egg/lxml/etree.so > #1 0xb76fbb12 in __pyx_f_4lxml_5etree__receiveXSLTError () > from /usr/lib/python2.5/site-packages/lxml-2.1.1-py2.5-linux-i686.egg/lxml/etree.so > #2 0xb766aef9 in xsltPrintErrorContext () from /usr/lib/libxslt.so.1 > #3 0xb766b091 in xsltTransformError () from /usr/lib/libxslt.so.1 > #4 0xb768d434 in xsltValueOf () from /usr/lib/libxslt.so.1 > #5 0xb768a5ba in ?? () from /usr/lib/libxslt.so.1 > #6 0x0828d230 in ?? () > #7 0x082bb618 in ?? () > #8 0x0828d578 in ?? () > #9 0x085db390 in ?? () > #10 0x00000000 in ?? () Hmmm, I have no idea where in the (pretty short) _forwardError() function it could crash, especially after having run correctly before. Also, I'm wondering why the later stack frames do not show up. Are you sure libxslt is actually called from lxml.etree here? Given your valgrind log, you seem to be using other XML libraries as well. Does any of them use libxslt independently? > I've attached the valgrind log, hopefully it's not too large for the > mailing list. It was. :) I won't forward it, since it's not really helpful to other readers. > Do you want the valgrind core file too? It's 150MB. I'm > trying to narrow down what I need to do first to get it to crash, I'll > let you know if I come up with a simple test case as a result. Please check any other libraries for libxslt usage first. The error log setup of lxml.etree is global for each thread, that might already be the problem here. If it is, it *might* be possible to remove the global setup at least for libxslt, but I will have to look into that. To figure out which library interferes here, you can install the debug packages for each library you use in the setup, until you get a complete stack trace above. Stefan

2 9

[lxml-dev] schema validation support
by Santoro, Peter Aug. 10, 2008

Aug. 10, 2008

I have used xerces/java, in the past, to do xml work without schema validation. As I now prefer to use python, I'm considering using lxml to validate xml instance documents against associated schemas. So far, I'm impressed with my initial testing. Thank you for making this software available! I'm curious if anyone has compared/benchmarked lxml/libxml2 schema validation support with what's available in Apache's xerces project? What, if any, limitations to the xlml/libxml2 schema validation support have others hit that I should be aware of? What, if any, limitations to the xerces schema validation support have others hit that I should be aware of? I'm asking these questions, because I recently noticed on the underlying libxml2 library's web site states that the xml schema support is incomplete (http://xmlsoft.org/index.html) and that it is being finished up (http://xmlsoft.org/news.html). To be fair, it appears that there are limitations with xerces schema validation support (http://xerces.apache.org/xerces-c/schema.html), too. Thank you, Peter

2 1