Mailman 3 February 2007 - lxml - The Python XML Toolkit

[lxml-dev] lxml 1.2 released
by Stefan Behnel Feb. 20, 2007

Feb. 20, 2007

Hi all, after a period of reduced activity, lxml 1.2 has finally been released. This is a somewhat conservative release in that it brings no major new features. It rather contains a number of bug fixes and cleanups, both internally and at the API level. Building lxml should have become easier again, and hacking the build process should now be a lot simpler. The complete changelog follows. Have fun, Stefan ========== ChangeLog: ========== 1.2 (2007-02-20) ================ Features added -------------- * Rich comparison of QName objects * Support for regular expressions in benchmark selection * get/set emulation (not .attrib!) for attributes on processing instructions * ElementInclude Python module for ElementTree compatible XInclude processing that honours custom resolvers registered with the source document * ElementTree.parser property holds the parser used to parse the document * setup.py has been refactored for greater readability and flexibility * --rpath flag to setup.py to induce automatic linking-in of dynamic library runtime search paths has been renamed to --auto-rpath. This makes it possible to pass an --rpath directly to distutils; previously this was being shadowed. Bugs fixed ---------- * Element instantiation now uses locks to prevent race conditions with threads * ElementTree.write() did not raise an exception when the file wasn't writable * Error handling could crash under Python <= 2.4.1 - fixed by disabling thread support in these environments * Element.find*() did not accept QName objects as path Other changes ------------- * code cleanup: redundant _NodeBase super class merged into _Element class Note: although the impact should be zero in most cases, this change breaks the compatibiliy of the public C-API

1 0

[lxml-dev] HTMLParser ignoring the namespaces
by Alain Poirier Feb. 16, 2007

Feb. 16, 2007

I've got a problem with the HTMLParser and namespaces. The XMLParser is fine : >>> from lxml import etree as ET >>> xml = ET.XML('') >>> for element in xml.getiterator(): >>> print element, element.attrib, element.nsmap <Element p at b7b032ac> {} {'foo': 'bar'} <Element p at b7b032d4> {'{bar}id': 'x'} {'foo': 'bar'} But with the HTMLParser, the nsmap properties are always empty : >>> from lxml import etree as ET >>> html = ET.HTML('') >>> for element in html.getiterator(): >>> print element, element.attrib, element.nsmap <Element html at b7b03324> {} {} <Element body at b7b0334c> {} {} <Element p at b7b03374> {'foo': 'bar'} {} <Element p at b7b0339c> {'id': 'x'} {} Any ideas ? -- Alain POIRIER

2 1

[lxml-dev] Objectify nodes can't have real attributes
by Tres Seaver Feb. 16, 2007

Feb. 16, 2007

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 For a project I'm currently working on, I want to create nodes by parsing an XML document, and then bind a Zope3 interface to the node, in order to allow for compmonent lookup. With classic elementtree nodes, and with lxml.etree nodes, I can. However, the nodes produced by lxml.objectify don't allow this. Following is a doctest which demonstrates what I'd like (the last stanza contains the tests which break). First, set up a marker interface:: >>> from zope.interface import Interface, directlyProvides >>> class IFoo(Interface): ... pass Now, test that we can mark an elementtree node:: >>> from elementtree.ElementTree import XML >>> node = XML('<node/>') >>> IFoo.providedBy(node) False >>> directlyProvides(node, IFoo) >>> IFoo.providedBy(node) True Now test an lxml.etree node:: >>> from lxml.etree import XML >>> node = XML('<node/>') >>> IFoo.providedBy(node) False >>> directlyProvides(node, IFoo) Traceback (most recent call last): ... AttributeError: 'etree._Element' object has no attribute '__provides__' OK, so we need to override the node class used by the parser:: >>> from lxml.etree import ElementBase >>> from zope.interface import implements >>> class MyNode(ElementBase): ... implements(IFoo) >>> from lxml.etree import XMLParser, ElementDefaultClassLookup >>> lookup = ElementDefaultClassLookup(element=MyNode) >>> parser = XMLParser() >>> parser.setElementClassLookup(lookup) >>> node = XML('<node/>', parser) >>> isinstance(node, MyNode) True >>> IFoo.providedBy(node) True Before we stamp the node with a custom interface, its interfaces come from its class:: >>> node.__provides__ <implementedBy __builtin__.MyNode> >>> before = node.__provides__ >>> list(before) [<InterfaceClass __builtin__.IFoo>] Afterwards, they are stored on the instance:: >>> class IBar(Interface): ... pass >>> IBar.providedBy(node) False >>> directlyProvides(node, IBar) >>> IBar.providedBy(node) True >>> node.__provides__ # doctest: +ELLIPSIS <zope.interface.Provides object at ...> >>> after = node.__provides__ >>> list(after) [<InterfaceClass __builtin__.IBar>, <InterfaceClass __builtin__.IFoo>] Now, let's try with lxml's objectify nodes:: >>> from lxml.objectify import XML >>> from lxml.objectify import ObjectifiedElement >>> node = XML('<node/>') >>> IFoo.providedBy(node) False In this case, we can *call* ``directlyProvides``, but it doesn't do what we want: instead of binding the interface, it creates a child node!:: >>> directlyProvides(node, IFoo) >>> type(node.__provides__) <type 'objectify.StringElement'> As with etree nodes we need to override the node class used by the parser:: >>> from lxml.etree import XML # objectify version won't take a parser >>> from lxml.objectify import ObjectifiedElement >>> class MyTreeNode(ObjectifiedElement): ... implements(IFoo) >>> from lxml.objectify import ObjectifyElementClassLookup >>> lookup = ObjectifyElementClassLookup(tree_class=MyTreeNode) >>> parser = XMLParser(remove_blank_text=True) >>> parser.setElementClassLookup(lookup) >>> node = XML('<node/>', parser) >>> node.__provides__ <implementedBy __builtin__.MyTreeNode> >>> IFoo.providedBy(node) True However, we still can't assign correctly to ``__provides__``:: >>> class IBar(Interface): ... pass >>> IBar.providedBy(node) False >>> before = node.__provides__ >>> list(before) [<InterfaceClass __builtin__.IFoo>] >>> directlyProvides(node, IBar) >>> after = node.__provides__ However, both these tests break:: >>> list(after) [<InterfaceClass __builtin__.IBar>, <InterfaceClass __builtin__.IFoo>] >>> IBar.providedBy(node) True We might try using a node class which has a slot for ``__provides__``:: >>> class MySlottedTreeNode(ObjectifiedElement): ... __slots__ = ('__provides__',) ... implements(IFoo) >>> lookup = ObjectifyElementClassLookup(tree_class=MyTreeNode) >>> parser = XMLParser(remove_blank_text=True) >>> parser.setElementClassLookup(lookup) >>> node = XML('<node/>', parser) >>> node.__provides__ <implementedBy __builtin__.MyTreeNode> >>> IFoo.providedBy(node) True >>> IBar.providedBy(node) False >>> before = node.__provides__ >>> list(before) [<InterfaceClass __builtin__.IFoo>] >>> directlyProvides(node, IBar) >>> after = node.__provides__ However, still no joy: both these break:: >>> list(after) [<InterfaceClass __builtin__.IBar>, <InterfaceClass __builtin__.IFoo>] >>> IBar.providedBy(node) True Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver(a)palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF0nL7+gerLs4ltQ4RAlmLAJ9cAykd6TUiX128agDpT7PBI+FOGACaA8P1 /WcuQLZcCaZuddFt4IV0Gok= =SLMx -----END PGP SIGNATURE-----

2 4

[lxml-dev] circular reference in element tree
by Holger Joukl Feb. 14, 2007

Feb. 14, 2007

Hi, here is another piece of the puzzle, hunting down a segfault/bus error problem: This time, my program did not core dump but seemed to hang in an endless loop in an ElementDepthFirstIterator: (gdb) where #0 0xfe0a2f3c in __pyx_f_5etree__elementFactory (__pyx_v_doc=0x4053a0, __pyx_v_c_node=0xdef3e0) at src/lxml/etree.c:7120 #1 0xfe0a80ac in __pyx_f_5etree_25ElementDepthFirstIterator___next__ (__pyx_v_self=0x3ff8a0) at src/lxml/etree.c:9081 #2 0x3b35c in listextend (self=0xdb0558, b=0xdd6fd0) at Objects/listobject.c:825 #3 0x3d028 in list_init (self=0xdb0558, args=0x417e30, kw=0x0) at Objects/listobject.c:2376 #4 0x58220 in type_call (type=0x107f7c, args=0x417e30, kwds=0x0) at Objects/typeobject.c:435 #5 0x26028 in PyObject_Call (func=0xdb0558, arg=0x417e30, kw=0x0) at Objects/abstract.c:1795 #6 0x8a514 in do_call (func=0x107f7c, pp_stack=0xfd108fa0, na=-1, nk=4292144) at Python/ceval.c:3771 #7 0x88324 in call_function (pp_stack=0xfd108fa0, oparg=1) at Python/ceval.c:3586 #8 0x8565c in PyEval_EvalFrame (f=0xbfc488) at Python/ceval.c:2163 #9 0x86b14 in PyEval_EvalCodeEx (co=0x3a9de0, globals=0x0, locals=0xbfc488, args=0xea400, argcount=959488, kws=0xea400, kwcount=1, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:2736 #10 0x884e4 in fast_function (func=0x3b15b0, pp_stack=0xfd1091f0, n=5, na=959488, nk=1) at Python/ceval.c:3656 #11 0x8830c in call_function (pp_stack=0xfd1091f0, oparg=3) at Python/ceval.c:3584 #12 0x8565c in PyEval_EvalFrame (f=0xdfde88) at Python/ceval.c:2163 #13 0x86b14 in PyEval_EvalCodeEx (co=0x3a9ce0, globals=0x0, locals=0xdfde88, args=0xea400, argcount=959488, kws=0xea400, kwcount=1, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:2736 #14 0x884e4 in fast_function (func=0x3a9ef0, pp_stack=0xfd109440, n=5, na=959488, nk=1) at Python/ceval.c:3656 #15 0x8830c in call_function (pp_stack=0xfd109440, oparg=3) at Python/ceval.c:3584 #16 0x8565c in PyEval_EvalFrame (f=0xdef468) at Python/ceval.c:2163 #17 0x88458 in fast_function (func=0x3a9870, pp_stack=0x267f10, n=1, na=1, nk=4629184) at Python/ceval.c:3645 #18 0x8830c in call_function (pp_stack=0xfd109608, oparg=1) at Python/ceval.c:3584 #19 0x8565c in PyEval_EvalFrame (f=0x267db0) at Python/ceval.c:2163 #20 0x88458 in fast_function (func=0x3a96f0, pp_stack=0x251920, n=1, na=1, nk=4629184) at Python/ceval.c:3645 #21 0x8830c in call_function (pp_stack=0xfd1097d0, oparg=1) at Python/ceval.c:3584 #22 0x8565c in PyEval_EvalFrame (f=0x2517c0) at Python/ceval.c:2163 #23 0x86b14 in PyEval_EvalCodeEx (co=0x1ca860, globals=0x0, locals=0x2517c0, args=0x408b7c, argcount=1, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:2736 #24 0xda520 in function_call (func=0x1d2bb0, arg=0x408b70, kw=0x0) at Objects/funcobject.c:548 #25 0x26028 in PyObject_Call (func=0x1d2bb0, arg=0x408b70, kw=0x0) at Objects/abstract.c:1795 #26 0x2e088 in instancemethod_call (func=0x1d2bb0, arg=0x408b70, kw=0x0) at Objects/classobject.c:2447 #27 0x26028 in PyObject_Call (func=0x1d2bb0, arg=0x408b70, kw=0x0) at Objects/abstract.c:1795 #28 0x8794c in PyEval_CallObjectWithKeywords (func=0x1c61c0, arg=0x12f030, kw=0x0) at Python/ceval.c:3430 #29 0xb7120 in t_bootstrap (boot_raw=0xbe77a0) at ./Modules/threadmodule.c:434 (gdb) (gdb) up #1 0xfe0a80ac in __pyx_f_5etree_25ElementDepthFirstIterator___next__ (__pyx_v_self=0x3ff8a0) at src/lxml/etree.c:9081 9081 __pyx_2 = ((PyObject *)__pyx_f_5etree__elementFactory(__pyx_v_current_node->_doc,__pyx_v_c_node)); if (!__pyx_2) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 1562; goto __pyx_L1;} (gdb) #2 0x3b35c in listextend (self=0xdb0558, b=0xdd6fd0) at Objects/listobject.c:825 825 } (gdb) p *((struct LxmlNodeBase*)(b))->_c_node $70 = { _private = 0xdd6fd0, type = XML_ELEMENT_NODE, name = 0xc0a4b3 "TIMACT", children = 0xdf5b20, last = 0xdf5b20, parent = 0xc0e4b8, next = 0x0, prev = 0xdf5b20, doc = 0xc49aa0, ns = 0x0, content = 0x0, properties = 0x0, nsDef = 0x0, psvi = 0x0, line = 0, extra = 0 } (gdb) p self $71 = (PyListObject *) 0xdb0558 (gdb) down #1 0xfe0a80ac in __pyx_f_5etree_25ElementDepthFirstIterator___next__ (__pyx_v_self=0x3ff8a0) at src/lxml/etree.c:9081 9081 __pyx_2 = ((PyObject *)__pyx_f_5etree__elementFactory(__pyx_v_current_node->_doc,__pyx_v_c_node)); if (!__pyx_2) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 1562; goto __pyx_L1;} (gdb) p __pyx_v_current_node->_doc $72 = (struct LxmlDocument * ?) 0x4053a0 (gdb) p *__pyx_v_current_node->_doc $73 = { ob_refcnt = 9, ob_type = 0xfe13a460, __pyx_vtab = 0xfe1474e0, _ns_counter = 0, _c_doc = 0xc49aa0, _parser = 0x298970 } (gdb) p *__pyx_v_c_node $74 = { _private = 0xdd6fd0, type = XML_ELEMENT_NODE, name = 0xc0a4b3 "TIMACT", children = 0xdf5b20, last = 0xdf5b20, parent = 0xc0e4b8, next = 0x0, prev = 0xdf5b20, doc = 0xc49aa0, ns = 0x0, content = 0x0, properties = 0x0, nsDef = 0x0, psvi = 0x0, line = 0, extra = 0 } (gdb) p *__pyx_v_c_node->children $75 = { _private = 0x0, type = XML_TEXT_NODE, name = 0xfdf9e050 "text", children = 0x0, last = 0x0, parent = 0xc0e4b8, next = 0xdef3e0, prev = 0x0, doc = 0xc49aa0, ns = 0x0, content = 0xe030c8 "09:26", properties = 0x0, nsDef = 0x0, psvi = 0x0, line = 0, extra = 0 } (gdb) p __pyx_v_c_node $76 = (xmlNode *) 0xdef3e0 (gdb) p *__pyx_v_c_node->children->next $77 = { _private = 0xdd6fd0, type = XML_ELEMENT_NODE, name = 0xc0a4b3 "TIMACT", children = 0xdf5b20, last = 0xdf5b20, parent = 0xc0e4b8, next = 0x0, prev = 0xdf5b20, doc = 0xc49aa0, ns = 0x0, content = 0x0, properties = 0x0, nsDef = 0x0, psvi = 0x0, line = 0, extra = 0 } (gdb) p (__pyx_v_c_node->children->next == __pyx_v_c_node) $78 = 1 (gdb) Note how the __pyx_v_c_node holds a reference to itself in __pyx_v_c_node->children->next (which I guess should never happen) How come this situation arises is currently a mystery to me. Although this is a threaded program now all my threads deepcopy any Elements they get presented, before doing anything with them. Still searching, Holger Der Inhalt dieser E-Mail ist vertraulich. Falls Sie nicht der angegebene Empfänger sind oder falls diese E-Mail irrtümlich an Sie adressiert wurde, verständigen Sie bitte den Absender sofort und löschen Sie die E-Mail sodann. Das unerlaubte Kopieren sowie die unbefugte Übermittlung sind nicht gestattet. Die Sicherheit von Übermittlungen per E-Mail kann nicht garantiert werden. Falls Sie eine Bestätigung wünschen, fordern Sie bitte den Inhalt der E-Mail als Hardcopy an. The contents of this e-mail are confidential. If you are not the named addressee or if this transmission has been addressed to you in error, please notify the sender immediately and then delete this e-mail. Any unauthorized copying and transmission is forbidden. E-Mail transmission cannot be guaranteed to be secure. If verification is required, please request a hard copy version. Landesbank Baden-Württemberg Anstalt des öffentlichen Rechts Hauptsitze: Stuttgart, Karlsruhe, Mannheim

2 1

Re: [lxml-dev] Build on AIX5, Python 2.3 support
by Stefan Behnel Feb. 13, 2007

Feb. 13, 2007

Hi, > Managed to build lxml 1.1.2 on AIX5.2, however I had to make a minor > patch to "setup.py" sorry for the late reply and thanks for the patch. It won't make it into the distribution, but it's good to have a solution to such a problem in the mailing list archive. Regarding the DocFileSuite, it's a test-suite-only problem that should be solved in current SVN. We added a local copy of a later doctest.py version to make this work (src/local_doctest.py). Python 2.3 is still supported and we will continue to support it as long as it makes sense. I've seen a Solaris system lately that came with Python 2.2 installed (which we can't possibly support), but 2.3 support is definitely in scope for us. Regards, Stefan

2 1

[lxml-dev] lxml 1.2 ahead
by Stefan Behnel Feb. 12, 2007

Feb. 12, 2007

Hi everyone, I will finally try to find some time within the next two weeks for releasing lxml 1.2 with the modular setup.py, a couple of bug fixes and a couple of enhancements. This is the current list of changes: http://codespeak.net/svn/lxml/trunk/CHANGES.txt It will *not* contain the rewritten namespace fixing code (nscleanup branch), which I couldn't get stable so far. There are still tests that fail, so it's not compatible enough to replace the original implementation. There has been a new series of Pyrex releases (0.9.5+), and I've written up a patch for it containing the public C-API stuff used by lxml. There is currently a problem with enums which are no longer considered ints by Pyrex. This is the only problem I see that keeps lxml from supporting (a patched) Pyrex 0.9.5+. Once that's solved, I'll make an updated version available from the SVN repository (/lxml/pyrex). I'm also still trying to get my patches finally merged into the mainstream version - we'll see... If there are any wishes for fixes or enhancements in lxml 1.2, now is a good time to speak up. Patches are appreciated, bigger things will have to wait for 1.3. Have fun, Stefan

2 1

[lxml-dev] Status of lxml 1.1.2
by Lee Brown Feb. 8, 2007

Feb. 8, 2007

Greetings! I have an application that would be greatly simplified by using the stylesheet-PI support in lxml 1.1.2 Has anyone made a Windows build of 1.1.2 yet? If so, where can I get it? Also, what was finally decided on as the API for xslt processing using a stylesheet-PI? I went back and re-read the mailing list traffic on the topic, but it wasn't clear to me how the final form of the API ended up. Best Regards, Lee E. Brown (leebrown(a)leebrown.org)

5 10

[lxml-dev] Validation against an external DTD
by mike＠it-loops.com Feb. 7, 2007

Feb. 7, 2007

Hello, I read through the documentation and I did not find a way to validate an XML-File against an external DTD with lxml. I searched the ML-archive and found several posts but I still do not know exactly, if this functionality is available or not. Right now I am using pyxml to do validation, but since the whole process will switch to xml schemes I am looking for a tool that can validate both (XSD,DTD). Is it possible to extend lxml to do this? I know that I can use a tool like "trang" to convert the DTDs but since there are DTDs that get added or removed from the process this means a lot of manual work. Kind regards, Michael

4 5