lxml - The Python XML Toolkit June 2008

lxml@python.org

29 participants
39 discussions

[lxml-dev] lxml 2.0.5 released
by Stefan Behnel 11 Jan '23

11 Jan '23

Hi all, lxml 2.0.5 is on PyPI. This is a bug-fix-only release of the stable 2.0 series. Have fun, Stefan 2.0.5 (2008-05-01) Bugs fixed * Resolving to a filename in custom resolvers didn't work. * lxml did not honour libxslt's second error state "STOPPED", which let some XSLT errors pass silently. * Memory leak in Schematron with libxml2 >= 2.6.31.

3 4

[lxml-dev] lxml has its page on launchpad
by Stefan Behnel 21 Sep '22

21 Sep '22

Hi all, I added the lxml project to launchpad, the Ubuntu Bug-Tracker. It also has a FAQ engine and a couple of other goodies. https://launchpad.net/lxml It's easy to sign up for launchpad, BTW, no 90%-footnotes-contract. Have fun, Stefan

3 2

[lxml-dev] Building LXML Trunk
by Sidnei da Silva 31 Aug '22

31 Aug '22

Hi, I've tried to build lxml from trunk today, on Win32. Got the following error: src\lxml\etree.c(880) : error C2059: syntax error : ')' src\lxml\etree.c(881) : error C2059: syntax error : ')' src\lxml\etree.c(882) : error C2059: syntax error : ')' src\lxml\etree.c(883) : error C2059: syntax error : ')' Any clue? Smells like a Pyrex issue? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

4 4

[lxml-dev] lxml 2.0 beta1 released
by Stefan Behnel 22 Dec '08

22 Dec '08

Hi all, I finally managed to push lxml 2.0beta1 over to PyPI. This release marks the end of the four month alpha cycle of lxml 2.0. The last stable release series, lxml 1.3, saw the light of day more than six months ago. http://codespeak.net/lxml/dev/ http://pypi.python.org/pypi/lxml/2.0beta1 The complete changelog for beta1 and the 2.0 alpha series follows below. Apart from a number of important fixes and enhancements, this beta release also finalises the major API changes that make the difference between 1.x and 2.x. Incompatible changes after this release will require a very good motivation. As usual, compatible enhancements will always be embraced - as will be updates, clarifications and fixes for the documentation! Asking back helps. I expect beta1 to also be the last beta release before lxml 2.0 final (hopefully not in the sense that alpha4/5/6 were), so please test as much as you can to spot any remaining bugs and problems. Note that this release depends on a bug fix in Cython that will hopefully be released as Cython 0.9.6.11 in a couple of days. I attached the necessary patch for those who want work on the sources. Another thing: there was a security advisory on the libxml2 mailing list. To prevent DoS attacks, systems that parse XML from untrusted sources should be updated to libxml2 2.6.31 (or should apply the patch that is referenced in Daniel's post below). http://mail.gnome.org/archives/xml/2008-January/msg00036.html Sidnei, when you build the Windows binaries, could you please wait for libxml2 2.6.31 to become available as binaries as well? Hopefully, that won't take too long... Have fun, Stefan 2.0beta1 (2008-01-11) ===================== Features added -------------- * Parse-time XML schema validation (``schema`` parser keyword). * XPath string results of the ``text()`` function and attribute selection make their Element container accessible through a ``getparent()`` method. As a side-effect, they are now always unicode objects (even ASCII strings). * ``XSLT`` objects are usable in any thread - at the cost of a deep copy if they were not created in that thread. * Invalid entity names and character references will be rejected by the ``Entity()`` factory. * ``entity.text`` returns the textual representation of the entity, e.g. ``&``. Bugs fixed ---------- * XPath on ElementTrees could crash when selecting the virtual root node of the ElementTree. * Compilation ``--without-threading`` was buggy in alpha5/6. Other changes ------------- * Minor performance tweaks for Element instantiation and subelement creation 2.0alpha6 (2007-12-19) ====================== Features added -------------- * New properties ``position`` and ``code`` on ParseError exception (as in ET 1.3) Bugs fixed ---------- * Memory leak in the ``parse()`` function. * Minor bugs in XSLT error message formatting. * Result document memory leak in target parser. Other changes ------------- * Various places in the XPath, XSLT and iteration APIs now require keyword-only arguments. * The argument order in ``element.itersiblings()`` was changed to match the order used in all other iteration methods. The second argument ('preceding') is now a keyword-only argument. * The ``getiterator()`` method on Elements and ElementTrees was reverted to return an iterator as it did in lxml 1.x. The ET API specification allows it to return either a sequence or an iterator, and it traditionally returned a sequence in ET and an iterator in lxml. However, it is now deprecated in favour of the ``iter()`` method, which should be used in new code wherever possible. * The 'pretty printed' serialisation of ElementTree objects now inserts newlines at the root level between processing instructions, comments and the root tag. * A 'pretty printed' serialisation is now terminated with a newline. * Second argument to ``lxml.etree.Extension()`` helper is no longer required, third argument is now a keyword-only argument ``ns``. * ``lxml.html.tostring`` takes an ``encoding`` argument. 2.0alpha5 (2007-11-24) ====================== Features added -------------- * Rich comparison of ``element.attrib`` proxies. * ElementTree compatible TreeBuilder class. * Use default prefixes for some common XML namespaces. * ``lxml.html.clean.Cleaner`` now allows for a ``host_whitelist``, and two overridable methods: ``allow_embedded_url(el, url)`` and the more general ``allow_element(el)``. * Extended slicing of Elements as in ``element[1:-1:2]``, both in etree and in objectify * Resolvers can now provide a ``base_url`` keyword argument when resolving a document as string data. * When using ``lxml.doctestcompare`` you can give the doctest option ``NOPARSE_MARKUP`` (like ``# doctest: +NOPARSE_MARKUP``) to suppress the special checking for one test. Bugs fixed ---------- * Target parser failed to report comments. * In the ``lxml.html`` ``iter_links`` method, links in ``<object>`` tags weren't recognized. (Note: plugin-specific link parameters still aren't recognized.) Also, the ``<embed>`` tag, though not standard, is now included in ``lxml.html.defs.special_inline_tags``. * Using custom resolvers on XSLT stylesheets parsed from a string could request ill-formed URLs. * With ``lxml.doctestcompare`` if you do ``<tag xmlns="...">`` in your output, it will then be namespace-neutral (before the ellipsis was treated as a real namespace). Other changes ------------- * The module source files were renamed to "lxml.*.pyx", such as "lxml.etree.pyx". This was changed for consistency with the way Pyrex commonly handles package imports. The main effect is that classes now know about their fully qualified class name, including the package name of their module. * Keyword-only arguments in some API functions, especially in the parsers and serialisers. 2.0alpha4 (2007-10-07) ====================== Features added -------------- Bugs fixed ---------- * AttributeError in feed parser on parse errors Other changes ------------- * Tag name validation in lxml.etree (and lxml.html) now distinguishes between HTML tags and XML tags based on the parser that was used to parse or create them. HTML tags no longer reject any non-ASCII characters in tag names but only spaces and the special characters ``<>&/"'``. 2.0alpha3 (2007-09-26) ====================== Features added -------------- * Separate ``feed_error_log`` property for the feed parser interface. The normal parser interface and ``iterparse`` continue to use ``error_log``. * The normal parsers and the feed parser interface are now separated and can be used concurrently on the same parser instance. * ``fromstringlist()`` and ``tostringlist()`` functions as in ElementTree 1.3 * ``iterparse()`` accepts an ``html`` boolean keyword argument for parsing with the HTML parser (note that this interface may be subject to change) * Parsers accept an ``encoding`` keyword argument that overrides the encoding of the parsed documents. * New C-API function ``hasChild()`` to test for children * ``annotate()`` function in objectify can annotate with Python types and XSI types in one step. Accompanied by ``xsiannotate()`` and ``pyannotate()``. Bugs fixed ---------- * XML feed parser setup problem * Type annotation for unicode strings in ``DataElement()`` Other changes ------------- * lxml.etree now emits a warning if you use XPath with libxml2 2.6.27 (which can crash on certain XPath errors) * Type annotation in objectify now preserves the already annotated type by default to prevent loosing type information that is already there. 2.0alpha2 (2007-09-15) ====================== Features added -------------- * ``ET.write()``, ``tostring()`` and ``tounicode()`` now accept a keyword argument ``method`` that can be one of 'xml' (or None), 'html' or 'text' to serialise as XML, HTML or plain text content. * ``iterfind()`` method on Elements returns an iterator equivalent to ``findall()`` * ``itertext()`` method on Elements * Setting a QName object as value of the .text property or as an attribute will resolve its prefix in the respective context * ElementTree-like parser target interface as described in http://effbot.org/elementtree/elementtree-xmlparser.htm * ElementTree-like feed parser interface on XMLParser and HTMLParser (``feed()`` and ``close()`` methods) Bugs fixed ---------- * lxml failed to serialise namespace declarations of elements other than the root node of a tree * Race condition in XSLT where the resolver context leaked between concurrent XSLT calls Other changes ------------- * ``element.getiterator()`` returns a list, use ``element.iter()`` to retrieve an iterator (ElementTree 1.3 compatible behaviour) 2.0alpha1 (2007-09-02) ====================== Features added -------------- * Reimplemented ``objectify.E`` for better performance and improved integration with objectify. Provides extended type support based on registered PyTypes. * XSLT objects now support deep copying * New ``makeSubElement()`` C-API function that allows creating a new subelement straight with text, tail and attributes. * XPath extension functions can now access the current context node (``context.context_node``) and use a context dictionary (``context.eval_context``) from the context provided in their first parameter * HTML tag soup parser based on BeautifulSoup in ``lxml.html.ElementSoup`` * New module ``lxml.doctestcompare`` by Ian Bicking for writing simplified doctests based on XML/HTML output. Use by importing ``lxml.usedoctest`` or ``lxml.html.usedoctest`` from within a doctest. * New module ``lxml.cssselect`` by Ian Bicking for selecting Elements with CSS selectors. * New package ``lxml.html`` written by Ian Bicking for advanced HTML treatment. * Namespace class setup is now local to the ``ElementNamespaceClassLookup`` instance and no longer global. * Schematron validation (incomplete in libxml2) * Additional ``stringify`` argument to ``objectify.PyType()`` takes a conversion function to strings to support setting text values from arbitrary types. * Entity support through an ``Entity`` factory and element classes. XML parsers now have a ``resolve_entities`` keyword argument that can be set to False to keep entities in the document. * ``column`` field on error log entries to accompany the ``line`` field * Error specific messages in XPath parsing and evaluation NOTE: for evaluation errors, you will now get an XPathEvalError instead of an XPathSyntaxError. To catch both, you can except on ``XPathError`` * The regular expression functions in XPath now support passing a node-set instead of a string * Extended type annotation in objectify: new ``xsiannotate()`` function * EXSLT RegExp support in standard XPath (not only XSLT) Bugs fixed ---------- * lxml.etree did not check tag/attribute names * The XML parser did not report undefined entities as error * The text in exceptions raised by XML parsers, validators and XPath evaluators now reports the first error that occurred instead of the last * Passing '' as XPath namespace prefix did not raise an error * Thread safety in XPath evaluators Other changes ------------- * objectify.PyType for None is now called "NoneType" * ``el.getiterator()`` renamed to ``el.iter()``, following ElementTree 1.3 - original name is still available as alias * In the public C-API, ``findOrBuildNodeNs()`` was replaced by the more generic ``findOrBuildNodeNsPrefix`` * Major refactoring in XPath/XSLT extension function code * Network access in parsers disabled by default

4 9

[lxml-dev] namespace strangeness in lxml 1.1
by Eric Jahn 24 Aug '08

24 Aug '08

Using python-lxml 1.1.1-1 which comes with Debian Etch, I am noticing some odd behavior in how lxml handles attribute namespaces, and namespaces in general. In the following code, all actual namespace urls should be mapped to namespaces in the resulting output, and they are, except if an attribute is involved: type="{http://domain2.info}someattribute Is this an lxml/libxml2 bug? The second odd behavior is that, for subelements, all the namespace declarations are automatically redeclared in the subelement tag, which is redundant. Does anyone know how to get around these problems? from lxml import etree NS1_NAMESPACE = "http://domain1.info" NS2_NAMESPACE = "http://domain2.info" NS1 = "{%s}" % NS1_NAMESPACE NS2 = "{%s}" % NS2_NAMESPACE NSMAP = {"NS1" : NS1_NAMESPACE , "NS2" : NS2_NAMESPACE} root = etree.Element(NS1 + "firstelement", nsmap=NSMAP) element = etree.Element(NS2 + "secondelement", nsmap=NSMAP, type = NS2 + "someattribute") root.append(element) print etree.tostring(root,pretty_print=True) Which results in the following output: <NS1:firstelement xmlns:NS1="http://domain1.info" xmlns:NS2="http://domain2.info"> <NS2:secondelement xmlns:NS1="http://domain1.info" xmlns:NS2="http://domain2.info" type="{http://domain2.info}someattribute"/> </NS1:firstelement>

2 7

[lxml-dev] XSLT and threading
by Martijn Faassen 30 Jul '08

30 Jul '08

Hi there, We've been trying to do XSLT transformations in a multi-threaded (Zope 2) situation. On 1.3.6 this won't work, but 2.0.x is supposed to have support for this. Unfortunately we're getting memory errors from within XSLT.__call__, and we think this is the problem: transform_ctxt = xslt.xsltNewTransformContext(self._c_style, c_doc) if transform_ctxt is NULL: _destroyFakeDoc(input_doc._c_doc, c_doc) python.PyErr_NoMemory() For some reason this sometimes works, sometimes fails; it doesn't always do this, so we suspect perhaps we're already in a copied XSLT sheet (by _copyXSLT) when this happens. We're also wondering about the threading strategy of 2.1.x; the copyXSLT code is removed. Is there a new strategy? I couldn't find anything about it in CHANGES.txt. I mean, I wouldn't be unhappy with a new strategy as actually re-parsing the stylesheet each time this gets called from a different thread is rather expensive (the stylesheet isn't cached as far as I can see). What's the new strategy, if any? (Unfortunately 2.1.x also gives us an error, though a different one. I don't have this error handy here.) We're trying to reduce this to a simpler test case that demonstrates the problem but we're having a hard time so far. Any hints would be welcome. Regards, Martijn

4 13

[lxml-dev] validation with multiple XSD files
by Arye 07 Jul '08

07 Jul '08

Hello all, I would like to so some schema validation and started with the instructions in : http://codespeak.net/lxml/dev/validation.html#xmlschema This all works great. Now I would like to extend this to a XSD file that includes many other files. In other words I have a directory of XSD files that I would like to use. The include statement look like this (the included file is referenced by its name): <?xml version="1.0"?> <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified"> <xsd:include schemaLocation="base.xsd"/> <xsd:element name="Price"> ... ... some types defined in "base.xsd" are used here I am new to lxml so sorry in advance if the question does not make sense. Regards, Arye.

3 4

[lxml-dev] objectify.deannotate: call to etree.cleanup_namespaces in 2.1beta
by Holger Joukl 01 Jul '08

01 Jul '08

Hi, I have a usecase where I need to deannotate an objectified tree and then manually set py:pytype or xsi:type attributes. However, this seems to be getting difficult with 2.1beta as deannotate wipes out all nsmap information with its call to cleanup_namespaces(), and I cannot set a namespaced attribute through <elt>.set(...) Maybe I'm missing something and there's a convenient way to do this? If not, could we make the call to cleanup_namespaces optional (defaults to True) in deannotate()? Holger -- GMX startet ShortView.de. Hier findest Du Leute mit Deinen Interessen! Jetzt dabei sein: http://www.shortview.de/?mc=sv_ext_mf@gmx

3 4

[lxml-dev] Segmentation fault in lxml.html after pickling
by Marcel Hellkamp 01 Jul '08

01 Jul '08

This script crashes with a segmentation fault :/ Using Python-2.5.2, libxslt-1.1.9, libxml2-2.6.32, lxml-2.1beta2, linux-i686 #!/usr/bin/python # coding=utf-8 import cPickle import lxml, lxml.html html = '''<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "DTD/xhtml1-strict.dtd"> <?xml version="1.0" encoding="UTF-8"?> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <title>Test Page</title> </head> <body> Test Page </body> </html>''' tree = lxml.html.fromstring(html) cf = open('test.pcl', 'w') cPickle.dump(tree, cf, -1) cf.close() cf = open('test.pcl', 'r') pickled_tree = cPickle.load(cf) cf.close() print 'This works fine...' lxml.html.tostring(tree) print 'This crashes...' lxml.html.tostring(pickled_tree)

4 6

Re: [lxml-dev] schema validation - what would be valid at this point?
by John Lovell 30 Jun '08

30 Jun '08

So with schema snippets that look like this, what would it give you as valid input options? <xs:attribute name="RefId" use="required" type="sif:RefIdType" /> <xs:simpleType name="RefIdType"> <xs:restriction base="sif:GUIDType" /> </xs:simpleType> <xs:simpleType name="GUIDType"> <xs:restriction> <xs:simpleType> <xs:restriction base="xs:token"> <xs:pattern value="[0-9A-F]{32}" xmlns:xs="http://www.w3.org/2001/XMLSchema" /> </xs:restriction> </xs:simpleType> </xs:restriction> </xs:simpleType> John W. Lovell Web Applications Engineer Northwest Educational Service District 1601 R Avenue Anacortes, WA 98221 (360) 299-4086 jlovell(a)nwesd.org www.nwesd.org Together We Can ... -----Original Message----- From: Terry Brown [mailto:tbrown@nrri.umn.edu] Sent: Monday, June 30, 2008 12:23 PM To: John Lovell Subject: Re: [lxml-dev] schema validation - what would be valid at this point? On Mon, 30 Jun 2008 11:49:50 -0700 "John Lovell" <jlovell(a)esd189.org> wrote: > Can you give an example of what that might look like? What I'd really like to have is the functionality of Emac's nxml-mode via a Python API. That mode parses the text up to the insert point, and can then tell you what elements would be valid there, according to any RELAX-NX schema you specify. So really I guess it makes more sense to be able to get the answer at parse time, rather than when the tree's complete. If, so far, the parser has seen "<html><head>" the answer to the question "what's valid at this point?" would be "base, isindex, link, meta, script, style, title". I don't know if the libxml2 schema API will answer that question, if not then it's more a libxml2 feature request (initally :) rather than an lxml thing. Just using HTML as an example, you could answer in terms of any user supplied schema. Sorry my initial description was confused because I was thinking about already complete and valid trees to which you might add something, it would be nice to handle that case too, but it seems to make more sense to approach it from the incomplete, half parsed :-) tree point of view first. Cheers -Terry > John W. Lovell > Web Applications Engineer > Northwest Educational Service District > 1601 R Avenue > Anacortes, WA 98221 > (360) 299-4086 > jlovell(a)nwesd.org > > www.nwesd.org > Together We Can ... > > > -----Original Message----- > From: lxml-dev-bounces(a)codespeak.net > [mailto:lxml-dev-bounces@codespeak.net] On Behalf Of Terry Brown > Sent: Monday, June 30, 2008 11:20 AM > To: lxml-dev(a)codespeak.net > Subject: [lxml-dev] schema validation - what would be valid at this > point? > > Hi all, > > First thanks for lxml - it's great. > > If this is the wrong place for feature requests please point me > elsewhere... > > A feature that would be very useful for a number of applications, > particularly editors: the ability to ask "what attributes / elements > would be valid at this point?" where I guess a point would be > identified by an XPath or perhaps an Element object. > > I don't know if any of the libxml2 schema APIs expose this, if they > don't, that's probably where the change needs to be made first, but if > they do it would be great if lxml passed it on to python. > > Thanks, > > Terry > _______________________________________________ > lxml-dev mailing list > lxml-dev(a)codespeak.net > http://codespeak.net/mailman/listinfo/lxml-dev > ----------------------------------------------------------------------- Terry Brown Natural Resources Research Institute Research Associate 5013 Miller Trunk Highway tbrown(a)nrri.umn.edu University of Minnesota, Duluth Ph. 218 720 4345 Duluth, Minnesota 55811 Fax 218 720 4328 http://beaver.nrri.umn.edu/~tbrown/

2 1

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

lxml - The Python XML Toolkit June 2008