Mailman 3 July 2012 - lxml - The Python XML Toolkit

lxml 3.0 alpha1 released
by Stefan Behnel 31 Jul '12

31 Jul '12

Hi everyone, I'm proud to announce the first alpha release of lxml 3.0. This release features initial support for PyPy and a new tree API for DTDs (by Walter Dörwald), as well as some major cleanups in the code base. Originally, it was targeted as lxml 2.4, but some major improvements as well as a couple of potentially incompatible changes make "3.0" appear more appropriate. Since this is an alpha release, it is not available from PyPI. You can get it from here: http://lxml.de/3.0/ Download: http://lxml.de/files/lxml-3.0alpha1.tgz Signature: http://lxml.de/files/lxml-3.0alpha1.tgz.asc GitHub: https://github.com/lxml/lxml/commit/bf47a9747fd5fef3322c7b39445d27d6bd0db7d1 This release was built using a pre-release of Cython 0.17: https://github.com/cython/cython/commit/1a0ff5e2e0cc78f07f059db10ecd79db036… If you are interested in commercial support or customisations for the lxml package, please contact me directly. Have fun, Stefan 3.0alpha1 (2012-07-31) ====================== Features added -------------- * Initial support for building in PyPy (through cpyext). * DTD objects gained an API that allows read access to their declarations. * xpathgrep.py gained support for parsing line-by-line (e.g. from grep output) and for surrounding the output with a new root tag. * E-factory in lxml.builder accepts subtypes of known data types (such as string subtypes) when building elements around them. * Add support for the :root pseudo-class in cssselect. * Tree iteration and iterparse() with a selective tag argument supports passing a set of tags. Tree nodes will be returned by the iterators if they match any of the tags. Bugs fixed ---------- * The .find*() methods in lxml.objectify no longer use XPath internally, which makes them faster in many cases (especially when short circuiting after a single or couple of elements) and fixes some behavioural differences compared to lxml.etree. Note that this means that they no longer support arbitrary XPath expressions but only the subset that the ElementPath language supports. The previous implementation was also redundant with the normal XPath support, which can be used as a replacement. * el.find('*') could accidentally return a comment or processing instruction that happened to be in the wrong spot. (Same for the other .find*() methods.) * The error logging is less intrusive and avoids a global setup where possible. * Fixed undefined names in html5lib parser. * xpathgrep.py did not work in Python 3. * Element.attrib.update() did not accept an attrib of another Element as parameter. * For subtypes of ElementBase that make the .text or .tail properties immutable (as in objectify, for example), inserting text when creating Elements through the E-Factory feature of the class constructor would fail with an exception, stating that the text cannot be modified. Other changes -------------- * The code base was overhauled to properly use 'const' where the API of libxml2 and libxslt requests it. This also has an impact on the public C-API of lxml itself, as defined in etreepublic.pxd, as well as the provided declarations in the lxml/includes/ directory. Code that uses these declarations may have to be adapted. On the plus side, this fixes several C compiler warnings, also for user code, thus making it easier to spot real problems again. * The functionality of "lxml.cssselect" was moved into a separate PyPI package called "cssselect". To continue using it, you must install that package separately. The "lxml.cssselect" module is still available and provides the same interface, provided the "cssselect" package can be imported at runtime. * Element attributes passed in as an attrib dict or as keyword arguments are now sorted by (namespaced) name before being created to make their order predictable for serialisation and iteration. Note that adding or deleting attributes afterwards does not take that order into account, i.e. setting a new attribute appends it after the existing ones. * Several classes that are for internal use only were removed from the lxml.etree module dict: _InputDocument, _ResolverRegistry, _ResolverContext, _BaseContext, _ExsltRegExp, _IterparseContext, _TempStore, _ExceptionContext, __ContentOnlyElement, _AttribIterator, _NamespaceRegistry, _ClassNamespaceRegistry, _FunctionNamespaceRegistry, _XPathFunctionNamespaceRegistry, _ParserDictionaryContext, _FileReaderContext, _ParserContext, _PythonSaxParserTarget, _TargetParserContext, _ReadOnlyProxy, _ReadOnlyPIProxy, _ReadOnlyEntityProxy, _ReadOnlyElementProxy, _OpaqueNodeWrapper, _OpaqueDocumentWrapper, _ModifyContentOnlyProxy, _ModifyContentOnlyPIProxy, _ModifyContentOnlyEntityProxy, _AppendOnlyElementProxy, _SaxParserContext, _FilelikeWriter, _ParserSchemaValidationContext, _XPathContext, _XSLTResolverContext, _XSLTContext, _XSLTQuotedStringParam * Several internal classes can no longer be inherited from: _InputDocument, _ResolverRegistry, _ExsltRegExp, _ElementUnicodeResult, _IterparseContext, _TempStore, _AttribIterator, _ClassNamespaceRegistry, _XPathFunctionNamespaceRegistry, _ParserDictionaryContext, _FileReaderContext, _PythonSaxParserTarget, _TargetParserContext, _ReadOnlyPIProxy, _ReadOnlyEntityProxy, _OpaqueDocumentWrapper, _ModifyContentOnlyPIProxy, _ModifyContentOnlyEntityProxy, _AppendOnlyElementProxy, _FilelikeWriter, _ParserSchemaValidationContext, _XPathContext, _XSLTResolverContext, _XSLTContext, _XSLTQuotedStringParam, _XSLTResultTree, _XSLTProcessingInstruction

2 2

lxml 2.3.5 released
by Stefan Behnel 31 Jul '12

31 Jul '12

Hi, I just released lxml 2.3.5. It's a pure bug fix release. http://pypi.python.org/pypi/lxml/2.3.5 Bugs fixed * Crash when merging text nodes in element.remove(). * Crash in sax/target parser when reporting empty doctype. This release was built with Cython 0.15.1. Have fun, Stefan

1 0

segfault - etree.strip_tags and drop_tree() on adjacent elements
by Michal Bozon 30 Jul '12

30 Jul '12

hi, i get segfaults on following simple example: import lxml.html, lxml.etree doc = '<h1>title</h1> <b>foo</b> <p>boo</p>' tree = lxml.html.fromstring(doc) lxml.etree.strip_tags(tree, 'b') tree.xpath('//h1')[0].drop_tree() print(lxml.html.tostring(tree)) # segfault version info: lxml 2.3.4, libxml 2.7.8, libxslt 1.1.26 (archlinux x86_64) also lxml 2.2.8, on current debian squeeze i386 in both cases both in py2 / py3 environments btw, is mixing the lxml.html and lxml.etree api expected to work properly? regards, michal bozon

2 1

efficient way of grabbing all items with an ID?
by Nicholas Freymuth 30 Jul '12

30 Jul '12

I noticed etree.XMLID and etree.XMLDTDID, but how do i combine both? I was hoping to get all items with an id, whether the id is "id", or whatever it is defined as in the DTD. Thanks in advance

1 0

Updated lxml quick reference
by John W. Shipman 25 Jul '12

25 Jul '12

A New Mexico Tech Computer Center publication, "Python XML processing with lxml", is now revised and expanded: http://www.nmt.edu/tcc/help/pubs/pylxml/ The previous version was not terribly helpful in working on documents with multiple namespaces. The new version has an introductory section explaining how multi-namespace documents are serialized and how they appear in ElementTree form, including explanations of Clark notation, namespace prefix dictionaries, and searching for and building elements within a specific namespace. I hope you will find this useful. Please do not hesitate to forward any suggestions or comments to tcc-doc(a)nmt.edu. Best regards, John Shipman (tcc-doc(a)nmt.edu), Applications Specialist New Mexico Tech Computer Center, Speare 146, Socorro, NM 87801 (575) 835-5735, http://www.nmt.edu/~john ``Let's go outside and commiserate with nature.'' --Dave Farber

1 0

understanding xincludes in lxml
by Toni Ruottu 24 Jul '12

24 Jul '12

People, I am trying to process an xml document with lxml. Some parts of the document are included from other xml-files with XInclude. In general I do nothing special, and it just works. The included files appear magically as parts of the element tree. However, when the included documents have errors, lxml seems to silently drop them. One can simulate this by adding an extra </test> tag into an included xml document. How do I detect the failing includes? Where can I find the filename of the broken xml file? --Toni

2 2

mathml schema problem
by Tim Arnold 19 Jul '12

19 Jul '12

I'm trying to create xml elements from strings marked up as mathml, and I'm using the mathml3 schema as follows. I'm working in the same directory as the xsd files; I generate the schema object and the parser, but the parser complains about the entities in the marked-up string. What am I doing wrong? thanks, --Tim % ls mathml3-common.xsd* mathml3-presentation.xsd* mathml3.xsd* mathml3-content.xsd* mathml3-strict-content.xsd* % python Python 2.7.1 (r271:86832, Apr 5 2011, 13:19:14) [GCC 4.2.1 20070719 [FreeBSD]] on freebsd8 Type "help", "copyright", "credits" or "license" for more information. >>> from lxml import etree >>> xsd = 'mathml3.xsd' >>> schema = etree.XMLSchema(etree.parse(xsd)) >>> parser = etree.XMLParser(schema=schema) >>> m = '''<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"> <mstyle displaystyle="true"> <mo>=</mo> <mi>t</mi> <mo>−</mo> <mi>j</mi> </mstyle> </math>''' >>> etree.fromstring(m,parser) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "lxml.etree.pyx", line 2756, in lxml.etree.fromstring (src/lxml/lxml.etree.c:54657) File "parser.pxi", line 1578, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:82774) File "parser.pxi", line 1457, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:81572) File "parser.pxi", line 965, in lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:78242) File "parser.pxi", line 569, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:74498) File "parser.pxi", line 650, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:75389) File "parser.pxi", line 590, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:74722) lxml.etree.XMLSyntaxError: Entity 'minus' not defined, line 1, column 67 >>>

2 2

Parse-time injected XSD default attributes
by Robert Kirchgessner 14 Jul '12

14 Jul '12

I noticed with one of my schemas that although the default values for complexed types are being injected correctly, the default attributes for simpleContent extensions are ignored. Is this a bug or the intended functionallity? Thank you.

2 1

Set tree root
by Evgeny Turnaev 14 Jul '12

14 Jul '12

Hello. Is it ok to use _setroot() method of _ElementTree? If i set root element will part of tree above current root be freed? Here is my usage: original_doc = etree.fromstring('<bar/>') # savepoint foo = etree.Element("foo") foo.append("p") ... foo.append(original_doc) xslt_transform(foo) # now i want revert changes made after savepoint to original_tree # and discard all elements created after.. # is it a correct way? tree = foo.getroottree() tree._setroot(original_doc) I cant use deepcopy() of original_tree because the point is: i am trying to do modifications of original tree in xslt_transform with extension functions. If memory allocated by foo will not be freed after _setroot() - will it be freed after original_tree is out of scope? -- -------------------------------------------- Турнаев Евгений Викторович +7 906 875 09 43 --------------------------------------------

2 1

Getting XSLT Parameters from Stylesheet when imported
by Thomas Schraitle 09 Jul '12

09 Jul '12

Hi, thanks to the awesome lxml module, parsing and transforming XML is really easy in Python now. So a big thank you to all the developers for their work and contribution, I like lxml A LOT! :-) Hope this question hasn't been asked before, but haven't found an answer in the doc yet. Let's say, I have a customization layer of the DocBook stylesheets which looks like this: --------[ mydocbook.xsl ]-------- <xsl:stylesheet version="1.0" xmlns:h="http://www.w3.org/1999/xhtml" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"> <xsl:import href="http://docbook.sourceforge.net/release/xsl/current/docbook.xsl"/> <xsl:param name="admon.graphics" select="1"/> </xsl:stylesheet> -------------------------------- The docbook.xsl imports a whole tree of other stylesheets and are separated for better maintenance. The stylesheets also defines lots of useful parameters (in our case, usually stored in .../xhtml/param.xsl) which modifies the behavior of the transformation. There is also a default value of the admon.graphics is zero parameter set to 0 (zero). In my customization layer, I've overwritten this parameter and set it to "1". To get the above parameter, this can be done with little XPath magic: >>> xslttree = etree.parse("mydocbook.xsl") >>> params = xslttree.xpath("/xsl:*/xsl:param[@name='admon.graphics']", namespaces={"xsl":"http://www.w3.org/1999/XSL/Transform"}) So far, so good. However, if I disable the above parameter in the customization layer and want to get the original value (in our example "0") the above code doesn't work. Of couse, the xslttree just "sees" only the elements, but does not "resolve" any imports. The question is: How can I access the "imported" parameter anyhow? Do I need to traverse the whole import tree? Or is there any other preferred or recommended method or trick which I should use? Probably I need the equivalent of a xinclude() method for XSLT. ;) Any pointers or hints are much appreciated! :-) Thanks! -- Gruß/Regards, Thomas Schraitle

1 1