Mailman 3 March 2008 - lxml - The Python XML Toolkit

[lxml-dev] lxml has its page on launchpad
by Stefan Behnel 11 Apr '23

11 Apr '23

Hi all, I added the lxml project to launchpad, the Ubuntu Bug-Tracker. It also has a FAQ engine and a couple of other goodies. https://launchpad.net/lxml It's easy to sign up for launchpad, BTW, no 90%-footnotes-contract. Have fun, Stefan

9 9

[lxml-dev] Reparenting a node
by Lawrence Oluyede 30 Jan '23

30 Jan '23

I have a doc A and a doc B, I'd like to put a node extracted from A in the document B but I always get a ValueError: ValueError: Element is not a child of this node. I didn't find any "setparent" in the API. How can I do this? -- Lawrence, oluyede.org - neropercaso.it "It is difficult to get a man to understand something when his salary depends on not understanding it" - Upton Sinclair

3 2

[lxml-dev] Building LXML Trunk
by Sidnei da Silva 31 Aug '22

31 Aug '22

Hi, I've tried to build lxml from trunk today, on Win32. Got the following error: src\lxml\etree.c(880) : error C2059: syntax error : ')' src\lxml\etree.c(881) : error C2059: syntax error : ')' src\lxml\etree.c(882) : error C2059: syntax error : ')' src\lxml\etree.c(883) : error C2059: syntax error : ')' Any clue? Smells like a Pyrex issue? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

4 4

[lxml-dev] lxml 2.0 beta1 released
by Stefan Behnel 22 Dec '08

22 Dec '08

Hi all, I finally managed to push lxml 2.0beta1 over to PyPI. This release marks the end of the four month alpha cycle of lxml 2.0. The last stable release series, lxml 1.3, saw the light of day more than six months ago. http://codespeak.net/lxml/dev/ http://pypi.python.org/pypi/lxml/2.0beta1 The complete changelog for beta1 and the 2.0 alpha series follows below. Apart from a number of important fixes and enhancements, this beta release also finalises the major API changes that make the difference between 1.x and 2.x. Incompatible changes after this release will require a very good motivation. As usual, compatible enhancements will always be embraced - as will be updates, clarifications and fixes for the documentation! Asking back helps. I expect beta1 to also be the last beta release before lxml 2.0 final (hopefully not in the sense that alpha4/5/6 were), so please test as much as you can to spot any remaining bugs and problems. Note that this release depends on a bug fix in Cython that will hopefully be released as Cython 0.9.6.11 in a couple of days. I attached the necessary patch for those who want work on the sources. Another thing: there was a security advisory on the libxml2 mailing list. To prevent DoS attacks, systems that parse XML from untrusted sources should be updated to libxml2 2.6.31 (or should apply the patch that is referenced in Daniel's post below). http://mail.gnome.org/archives/xml/2008-January/msg00036.html Sidnei, when you build the Windows binaries, could you please wait for libxml2 2.6.31 to become available as binaries as well? Hopefully, that won't take too long... Have fun, Stefan 2.0beta1 (2008-01-11) ===================== Features added -------------- * Parse-time XML schema validation (``schema`` parser keyword). * XPath string results of the ``text()`` function and attribute selection make their Element container accessible through a ``getparent()`` method. As a side-effect, they are now always unicode objects (even ASCII strings). * ``XSLT`` objects are usable in any thread - at the cost of a deep copy if they were not created in that thread. * Invalid entity names and character references will be rejected by the ``Entity()`` factory. * ``entity.text`` returns the textual representation of the entity, e.g. ``&``. Bugs fixed ---------- * XPath on ElementTrees could crash when selecting the virtual root node of the ElementTree. * Compilation ``--without-threading`` was buggy in alpha5/6. Other changes ------------- * Minor performance tweaks for Element instantiation and subelement creation 2.0alpha6 (2007-12-19) ====================== Features added -------------- * New properties ``position`` and ``code`` on ParseError exception (as in ET 1.3) Bugs fixed ---------- * Memory leak in the ``parse()`` function. * Minor bugs in XSLT error message formatting. * Result document memory leak in target parser. Other changes ------------- * Various places in the XPath, XSLT and iteration APIs now require keyword-only arguments. * The argument order in ``element.itersiblings()`` was changed to match the order used in all other iteration methods. The second argument ('preceding') is now a keyword-only argument. * The ``getiterator()`` method on Elements and ElementTrees was reverted to return an iterator as it did in lxml 1.x. The ET API specification allows it to return either a sequence or an iterator, and it traditionally returned a sequence in ET and an iterator in lxml. However, it is now deprecated in favour of the ``iter()`` method, which should be used in new code wherever possible. * The 'pretty printed' serialisation of ElementTree objects now inserts newlines at the root level between processing instructions, comments and the root tag. * A 'pretty printed' serialisation is now terminated with a newline. * Second argument to ``lxml.etree.Extension()`` helper is no longer required, third argument is now a keyword-only argument ``ns``. * ``lxml.html.tostring`` takes an ``encoding`` argument. 2.0alpha5 (2007-11-24) ====================== Features added -------------- * Rich comparison of ``element.attrib`` proxies. * ElementTree compatible TreeBuilder class. * Use default prefixes for some common XML namespaces. * ``lxml.html.clean.Cleaner`` now allows for a ``host_whitelist``, and two overridable methods: ``allow_embedded_url(el, url)`` and the more general ``allow_element(el)``. * Extended slicing of Elements as in ``element[1:-1:2]``, both in etree and in objectify * Resolvers can now provide a ``base_url`` keyword argument when resolving a document as string data. * When using ``lxml.doctestcompare`` you can give the doctest option ``NOPARSE_MARKUP`` (like ``# doctest: +NOPARSE_MARKUP``) to suppress the special checking for one test. Bugs fixed ---------- * Target parser failed to report comments. * In the ``lxml.html`` ``iter_links`` method, links in ``<object>`` tags weren't recognized. (Note: plugin-specific link parameters still aren't recognized.) Also, the ``<embed>`` tag, though not standard, is now included in ``lxml.html.defs.special_inline_tags``. * Using custom resolvers on XSLT stylesheets parsed from a string could request ill-formed URLs. * With ``lxml.doctestcompare`` if you do ``<tag xmlns="...">`` in your output, it will then be namespace-neutral (before the ellipsis was treated as a real namespace). Other changes ------------- * The module source files were renamed to "lxml.*.pyx", such as "lxml.etree.pyx". This was changed for consistency with the way Pyrex commonly handles package imports. The main effect is that classes now know about their fully qualified class name, including the package name of their module. * Keyword-only arguments in some API functions, especially in the parsers and serialisers. 2.0alpha4 (2007-10-07) ====================== Features added -------------- Bugs fixed ---------- * AttributeError in feed parser on parse errors Other changes ------------- * Tag name validation in lxml.etree (and lxml.html) now distinguishes between HTML tags and XML tags based on the parser that was used to parse or create them. HTML tags no longer reject any non-ASCII characters in tag names but only spaces and the special characters ``<>&/"'``. 2.0alpha3 (2007-09-26) ====================== Features added -------------- * Separate ``feed_error_log`` property for the feed parser interface. The normal parser interface and ``iterparse`` continue to use ``error_log``. * The normal parsers and the feed parser interface are now separated and can be used concurrently on the same parser instance. * ``fromstringlist()`` and ``tostringlist()`` functions as in ElementTree 1.3 * ``iterparse()`` accepts an ``html`` boolean keyword argument for parsing with the HTML parser (note that this interface may be subject to change) * Parsers accept an ``encoding`` keyword argument that overrides the encoding of the parsed documents. * New C-API function ``hasChild()`` to test for children * ``annotate()`` function in objectify can annotate with Python types and XSI types in one step. Accompanied by ``xsiannotate()`` and ``pyannotate()``. Bugs fixed ---------- * XML feed parser setup problem * Type annotation for unicode strings in ``DataElement()`` Other changes ------------- * lxml.etree now emits a warning if you use XPath with libxml2 2.6.27 (which can crash on certain XPath errors) * Type annotation in objectify now preserves the already annotated type by default to prevent loosing type information that is already there. 2.0alpha2 (2007-09-15) ====================== Features added -------------- * ``ET.write()``, ``tostring()`` and ``tounicode()`` now accept a keyword argument ``method`` that can be one of 'xml' (or None), 'html' or 'text' to serialise as XML, HTML or plain text content. * ``iterfind()`` method on Elements returns an iterator equivalent to ``findall()`` * ``itertext()`` method on Elements * Setting a QName object as value of the .text property or as an attribute will resolve its prefix in the respective context * ElementTree-like parser target interface as described in http://effbot.org/elementtree/elementtree-xmlparser.htm * ElementTree-like feed parser interface on XMLParser and HTMLParser (``feed()`` and ``close()`` methods) Bugs fixed ---------- * lxml failed to serialise namespace declarations of elements other than the root node of a tree * Race condition in XSLT where the resolver context leaked between concurrent XSLT calls Other changes ------------- * ``element.getiterator()`` returns a list, use ``element.iter()`` to retrieve an iterator (ElementTree 1.3 compatible behaviour) 2.0alpha1 (2007-09-02) ====================== Features added -------------- * Reimplemented ``objectify.E`` for better performance and improved integration with objectify. Provides extended type support based on registered PyTypes. * XSLT objects now support deep copying * New ``makeSubElement()`` C-API function that allows creating a new subelement straight with text, tail and attributes. * XPath extension functions can now access the current context node (``context.context_node``) and use a context dictionary (``context.eval_context``) from the context provided in their first parameter * HTML tag soup parser based on BeautifulSoup in ``lxml.html.ElementSoup`` * New module ``lxml.doctestcompare`` by Ian Bicking for writing simplified doctests based on XML/HTML output. Use by importing ``lxml.usedoctest`` or ``lxml.html.usedoctest`` from within a doctest. * New module ``lxml.cssselect`` by Ian Bicking for selecting Elements with CSS selectors. * New package ``lxml.html`` written by Ian Bicking for advanced HTML treatment. * Namespace class setup is now local to the ``ElementNamespaceClassLookup`` instance and no longer global. * Schematron validation (incomplete in libxml2) * Additional ``stringify`` argument to ``objectify.PyType()`` takes a conversion function to strings to support setting text values from arbitrary types. * Entity support through an ``Entity`` factory and element classes. XML parsers now have a ``resolve_entities`` keyword argument that can be set to False to keep entities in the document. * ``column`` field on error log entries to accompany the ``line`` field * Error specific messages in XPath parsing and evaluation NOTE: for evaluation errors, you will now get an XPathEvalError instead of an XPathSyntaxError. To catch both, you can except on ``XPathError`` * The regular expression functions in XPath now support passing a node-set instead of a string * Extended type annotation in objectify: new ``xsiannotate()`` function * EXSLT RegExp support in standard XPath (not only XSLT) Bugs fixed ---------- * lxml.etree did not check tag/attribute names * The XML parser did not report undefined entities as error * The text in exceptions raised by XML parsers, validators and XPath evaluators now reports the first error that occurred instead of the last * Passing '' as XPath namespace prefix did not raise an error * Thread safety in XPath evaluators Other changes ------------- * objectify.PyType for None is now called "NoneType" * ``el.getiterator()`` renamed to ``el.iter()``, following ElementTree 1.3 - original name is still available as alias * In the public C-API, ``findOrBuildNodeNs()`` was replaced by the more generic ``findOrBuildNodeNsPrefix`` * Major refactoring in XPath/XSLT extension function code * Network access in parsers disabled by default

4 9

[lxml-dev] Pickling objectified trees
by Christian Zagrodnick 09 Jun '08

09 Jun '08

Hi, the other day I had to pickle objectified trees. I just thought to share my findings. Pickling is about serialization. IMHO the natural serialization of an objectified tree is its XML representation. So the following basically does that: -------------------------- import copy_reg import lxml.etree import lxml.objectify def treeFactory(state): """Un-Pickle factory.""" return lxml.objectify.fromstring(state) copy_reg.constructor(treeFactory) def reduceObjectifiedElement(object): """Reduce function for lxml.objectify trees. See http://docs.python.org/lib/pickle-protocol.html for details. """ return (treeFactory, (lxml.etree.tostring(object), )) copy_reg.pickle(lxml.objectify.ObjectifiedElement, reduceObjectifiedElement, treeFactory) ----------------------------------------- You might consider just registering the reduce function in lxml itself. Shouldn't hurt, should it. -- Christian Zagrodnick gocept gmbh & co. kg · forsterstrasse 29 · 06112 halle/saale www.gocept.com · fon. +49 345 12298894 · fax. +49 345 12298891

2 4

[lxml-dev] Help with an error message
by Konstantin Ryabitsev 14 May '08

14 May '08

Hi, everyone: I'm having trouble with the following case. One of my automatic import scripts takes data from one source and submits it to another as an XML feed. Recently, it started failing because one of the entries contains a null. The testcase is such: from lxml.etree import Element sourcestr = 'Contains a null: \x00' unistr = unicode(sourcestr, 'utf-8') elt = Element('foo').text = unistr Running it will cause the following error: Traceback (most recent call last): File "foo.py", line 6, in <module> elt = Element('foo').text = unistr File "etree.pyx", line 741, in etree._Element.text.__set__ File "apihelpers.pxi", line 344, in etree._setNodeText File "apihelpers.pxi", line 648, in etree._utf8 AssertionError: All strings must be XML compatible, either Unicode or ASCII Can someone suggest the best way to deal with this? Kind regards, -- Konstantin Ryabitsev Montréal, Québec

4 5

[lxml-dev] Can't load external DTD.
by Nef Asus 15 Apr '08

15 Apr '08

Hello everyone, I've written this little program that refuses to work: from lxml import etree if __name__ == "__main__": xml_input = "C:\Desarrollo\pythontests\lxml\foo.xml" parser = etree.XMLParser(load_dtd = True, dtd_validation = True, attribute_defaults = True) doc = etree.parse(xml_input, parser) Here's the traceback. Traceback (most recent call last): File "C:\Desarrollo\pythontests\lxml\dtd_loader.py", \ line 27, in <module> doc = etree.parse(xml_input, parser) File "lxml.etree.pyx", line 2515, in lxml.etree.parse File "parser.pxi", line 1755, in lxml.etree._parseDocument File "parser.pxi", line 1759, in lxml.etree._parseDocumentFromURL File "parser.pxi", line 1681, in lxml.etree._parseDocFromFile File "parser.pxi", line 826 ,in lxml.etree._BaseParser._parseDocFromFile File "parser.pxi",line 450,in lxml.etree._ParserContext._handleParseResultDoc File "parser.pxi", line 534, in lxml.etree._handleParseResult File "parser.pxi", line 476, in lxml.etree._raiseParseError lxml.etree.XMLSyntaxError: failed to load external entity "NULL", line 9, column 83 This is a snippet of foo.xml : <?xml version="1.0" encoding="iso-8859-1" ?> <!DOCTYPE rem:requirementsProject SYSTEM "C:\Desarrollo\pythontests\lxml\foo.dtd"> ... Then, I tried to write a custom resolver. from lxml import etree class DTDResolver(etree.Resolver): def resolve(self, url, id, context): print("Resolving (url, %s)(id, %s)"% (url,id)) self.resolve_filename("C:\Desarrollo\pythontests\lxml\JENSEN.dtd", \ context) if __name__ == "__main__": parser = etree.XMLParser(load_dtd = True, dtd_validation = True, attribute_defaults = True) parser.resolvers.add(DTDResolver()) xml_input = "C:\Desarrollo\pythontests\lxml\foo.xml" doc = etree.parse(xml_input, parser) But it still fails (same traceback). What am I doing wrong? BTW, lxml version is 2.0.1. Thanks in advance.

4 7

[lxml-dev] Help getting lxml to work reliably on MacOS-X
by Stefan Behnel 09 Apr '08

09 Apr '08

Hi, it looks like many Mac users have problems with lxml on their platform. This usually involves installing up-to-date dependencies (libxml2/libxslt) in addition to the system libraries. I would like to get these problems resolved. To get a start on this, we must collect some information. We had a few reports, but I need to know in more detail what people did, what they tried, and to what avail. So here is a list of questions for Mac users. Please help us by answering them. Some instructions follow at the end. When building lxml, please move any installed Cython versions out of the way and run the build on the unpacked lxml-2.0.tar.gz release sources. It must say "trying to build without Cython" at the beginning. Please provide the following information: - what package management system (fink/macports) do you use? - are you using the stock Python or one that is installed separately? - what library versions are you using of libxml2, libxslt, zlib, libiconv? - which library versions are preinstalled on your platform? (I do not know how to find that out, can anyone provide instructions here?) - what library versions does lxml.etree find? (see below) Just in case there are people who actually have a working installation, - has anyone successfully built lxml statically against libxml2/libxslt? * does it work reliably? (see "Testing" below) * did you build with the --without-threads option? * does it work with *and* without that option? - has anyone managed to get lxml working reliably (see "Testing" below) with a dynamic build? * did you set DYLD_LIBRARY_PATH? * is DYLD_LIBRARY_PATH required for you or does it work without? * is there anything special you did to make it work? * if there are crashes, is the install unusable or are there things you can still do reliably? If lxml crashes for you, - does it work if you set DYLD_LIBRARY_PATH at runtime? (dynamic builds only) - does it work when building with the --without-threads option? - does it crash when running the normal tests? - if the tests pass fine, does it crash with the "ot-test" script below? Every reply is appreciated. You can reply in private e-mail, if you prefer, although it might be helpful to others to see your answer. Thanks in advance, Stefan Here are some instructions: * Checking library versions Please report the library versions that lxml.etree thinks it is using: http://codespeak.net/lxml/FAQ.html#i-think-i-have-found-a-bug-in-lxml-what-… * Setting DYLD_LIBRARY_PATH When you run tests or an application with lxml, you can pass the environment variable DYLD_LIBRARY_PATH to your program. It needs to be set to the directory (or directories) where lxml's library dependencies are installed, i.e. libxml2 and libxslt. http://codespeak.net/lxml/build.html#providing-newer-library-versions-on-ma… * Testing When I say "works reliably", I mean, without crashes. The first thing to verify that is to run "make test". If it already crashes here, there is no need to try the script below. Please report anything you can find out about the crash in this case. A debugger run and/or a stack trace might give us some useful hints here. If the normal tests pass, please try another test. Here is an XML document and a script that fires off a bunch of threads to parse the document and run multiple XSLTs over it. The archive is about 1MB. http://codespeak.net/lxml/ot-test.zip It's kind of a worst-case scenario that I use to find problems, so if you have a working installation of lxml 2.0, please run it to see if it crashes for you. Parsing may fail (usually for threads 7-9), that's fine, but it must not crash with your otherwise working setup. It normally starts up 16 threads, which should require a couple of hundred MB of memory. If you find that it runs out of memory, you can try running less threads instead by passing the number to the script at the command line.

7 17

[lxml-dev] ElementTree.find does not accept QName objects.
by John Krukoff 06 Apr '08

06 Apr '08

Since I was the one that complained about the find method on Elements not accepting QNames, it's probably not surprising that I expected them to work with the ElementTree find method as well. Instead an unsliceable error is thrown, due to the value being expected to be a string: >>> from lxml import etree >>> x = etree.XML( '<a><b/></a>' ) >>> e = etree.ElementTree( x ) >>> e.find( 'b' ) <Element b at b7f4d914> >>> e.find( etree.QName( 'b' ) ) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "etree.pyx", line 508, in etree._ElementTree.find TypeError: 'etree.QName' object is unsliceable Though using e.getroot( ).find is working fine as a workaround for me. -- John Krukoff <jkrukoff(a)ltgc.com> Land Title Guarantee Company

2 4

Re: [lxml-dev] [BUG] lxml-2* hangs on interpreter shutdown with gtk-mainloop
by Stefan Behnel 27 Mar '08

27 Mar '08

Hi, René 'Necoro' Neumann wrote: >> I attached a patch that also works for me and (again) seems to work >> around the symptom you see. > > This patch only works for me when passing a filename or file to parse(). > When passing a StringIO instance it still hangs. True, didn't realise that in my tests. Here is another patch that seems to work better for me (still no idea why). > What about the way simpler patch: > Add the following lines somewhere, where they fit: > > if base_url is None: > base_url = "" > > Because passing an empty base_url works too. Or does this have > side-effects? Yes. The URL is used to identify the document in a couple of places, and the base_url parameter is just a way to override the URL in cases where lxml can't determine it itself. Stefan

2 7