Mailman 3 March 2009 - lxml - The Python XML Toolkit

[lxml-dev] test failures, lxml-2.2
by Jorge Barrios 26 Mar '09

26 Mar '09

I am hoping that there's an easy fix. I couldn't find anything about it in the mailing list. Anyone know what the problem might be? Jorge root@a0110:/data3/lxml-2.2# uname -a Linux a0110 2.6.18-8.el5PAE #1 SMP Thu Mar 15 20:29:51 EDT 2007 i686 i686 i386 GNU/Linux root@a0110:/data3/lxml-2.2# make test python setup.py build_ext -i Building lxml version 2.2. Building with Cython 0.11. Using build configuration of libxslt 1.1.24 Building against libxml2/libxslt in the following directory: /usr/ local/lib running build_ext python test.py -p -v TESTED VERSION: 2.2.0 Python: (2, 5, 2, 'final', 0) lxml.etree: (2, 2, 0, 0) libxml used: (2, 7, 3) libxml compiled: (2, 7, 3) libxslt used: (1, 1, 24) libxslt compiled: (1, 1, 24) 1010/1010 (100.0%): Doctest: xpathxslt.txt ---------------------------------------------------------------------- Ran 1010 tests in 29.973s OK PYTHONPATH=src: python selftest.py ********************************************************************** File "/var/data3/lxml-2.2/selftest.py", line 575, in selftest.encoding Failed example: serialize(elem) Expected: '<tag key="åöö<>"/>' Got: '<tag key="åöö<>"/>' ********************************************************************** File "/var/data3/lxml-2.2/selftest.py", line 577, in selftest.encoding Failed example: serialize(elem, encoding="utf-8") Expected: '<tag key="\xc3\xa5\xc3\xb6\xc3\xb6<>"/>' Got: '<tag key="åöö<>"/>' ********************************************************************** File "/var/data3/lxml-2.2/selftest.py", line 579, in selftest.encoding Failed example: serialize(elem, encoding="us-ascii") Expected: '<tag key="åöö<>"/>' Got: '<tag key="åöö<>"/>' ********************************************************************** File "/var/data3/lxml-2.2/selftest.py", line 581, in selftest.encoding Failed example: serialize(elem, encoding="iso-8859-1").lower() Expected: '<?xml version=\'1.0\' encoding=\'iso-8859-1\'?>\n<tag key="\xe5\xf6\xf6<>"/>' Got: '<?xml version=\'1.0\' encoding=\'iso-8859-1\'?>\n<tag key="åöö<>"/>' ********************************************************************** 1 items had failures: 4 of 29 in selftest.encoding ***Test Failed*** 4 failures. 181 tests ok. PYTHONPATH=src: python selftest2.py ********************************************************************** File "/var/data3/lxml-2.2/selftest2.py", line 196, in selftest2.encoding Failed example: serialize(elem) Expected: '<tag key="åöö<>"/>' Got: '<tag key="åöö<>"/>' ********************************************************************** File "/var/data3/lxml-2.2/selftest2.py", line 198, in selftest2.encoding Failed example: serialize(elem, "utf-8") Expected: '<tag key="\xc3\xa5\xc3\xb6\xc3\xb6<>"/>' Got: '<tag key="åöö<>"/>' ********************************************************************** File "/var/data3/lxml-2.2/selftest2.py", line 200, in selftest2.encoding Failed example: serialize(elem, "us-ascii") Expected: '<tag key="åöö<>"/>' Got: '<tag key="åöö<>"/>' ********************************************************************** File "/var/data3/lxml-2.2/selftest2.py", line 202, in selftest2.encoding Failed example: serialize(elem, "iso-8859-1").lower() Expected: '<?xml version=\'1.0\' encoding=\'iso-8859-1\'?>\n<tag key="\xe5\xf6\xf6<>"/>' Got: '<?xml version=\'1.0\' encoding=\'iso-8859-1\'?>\n<tag key="åöö<>"/>' ********************************************************************** 1 items had failures: 4 of 29 in selftest2.encoding ***Test Failed*** 4 failures. 102 tests ok. root@a0110:/data3/lxml-2.2#

2 1

[lxml-dev] Spacing and the presence of xml:space="preserve"
by F Wolff 24 Mar '09

24 Mar '09

Hallo all We are currently using this expression to obtain a plain text version inside a node: For example: >>> from lxml import etree >>> etree.XPath("string()") >>> string_xpath(etree.fromstring("<a> asdf <b/>fdsa </a>")) ' asdf fdsa ' This works great and returns the string assuming xml:space="preserve", in other words, spacing is taken verbatim. We work on a file format where some of the spacing is very important (XLIFF). We generate such files with xml:space="preserve" in the necessary places. Not everybody generates such files, unfortunately, so we need to also handle the normalised versions. If I rather use the XPath function "normalize-space()", I can get the normalised spacing: 'asdf fdsa' but unfortunately it does this even if xml:space="preserve" is set: >>> etree.XPath("normalize-space()") >>> string_xpath(etree.fromstring('''<a xml:space="preserve"> asdf <b/>fdsa </a>''')) 'asdf fdsa' Unfortunately, I don't see a way to get the correct version (normalised by default, but with white-space preserved if xml:space="preserved" is set). Do I have to handle the cases separately, or is there a way for lxml to help me by just doing the right thing? I could special case on the node, but it would be a bit harder to know if some xml:space directive was given higher up in the tree. Or am I missing something in XPath / lxml? Any help would be appreciated. Friedel Wolff -- Recently on my blog: http://translate.org.za/blogs/friedel/en/content/video-virtaals-functionali…

2 1

[lxml-dev] lxml 2.2 released
by Stefan Behnel 21 Mar '09

21 Mar '09

Hi all, I'm proud to announce the release of lxml 2.2 final. This is a major new, stable and mature release that takes over the stable 2.x release series. All previous 2.x releases are now officially out of maintenance. It includes a large number of bug fixes and improvements (see below for a complete changelog) that make lxml 2.2 a lot more robust than the previous 2.1 and older releases. It is therefore generally worth upgrading (and it should not be too hard to do that). You can get it from: http://codespeak.net/lxml/ http://pypi.python.org/pypi/lxml/2.2 This release was built with Cython 0.11 final. Have fun, Stefan 2.2 (2009-03-21) Features added * Support for standalone flag in XML declaration through tree.docinfo.standalone and by passing standalone=True/False on serialisation. Bugs fixed * Crash when parsing an XML Schema with external imports from a filename. 2.2beta4 (2009-02-27) Features added * Support strings and instantiable Element classes as child arguments to the constructor of custom Element classes. * GZip compression support for serialisation to files and file-like objects. Bugs fixed * Deep-copying an ElementTree copied neither its sibling PIs and comments nor its internal/external DTD subsets. * Soupparser failed on broken attributes without values. * Crash in XSLT when overwriting an already defined attribute using xsl:attribute. * Crash bug in exception handling code under Python 3. This was due to a problem in Cython, not lxml itself. * lxml.html.FormElement._name() failed for non top-level forms. * TAG special attribute in constructor of custom Element classes was evaluated incorrectly. Other changes * Official support for Python 3.0.1. * Element.findtext() now returns an empty string instead of None for Elements without text content. 2.2beta3 (2009-02-17) Features added * XSLT.strparam() class method to wrap quoted string parameters that require escaping. Bugs fixed * Memory leak in XPath evaluators. * Crash when parsing indented XML in one thread and merging it with other documents parsed in another thread. * Setting the base attribute in lxml.objectify from a unicode string failed. * Fixes following changes in Python 3.0.1. * Minor fixes for Python 3. Other changes * The global error log (which is copied into the exception log) is now local to a thread, which fixes some race conditions. * More robust error handling on serialisation. 2.2beta2 (2009-01-25) Bugs fixed * Potential memory leak on exception handling. This was due to a problem in Cython, not lxml itself. * iter_links (and related link-rewriting functions) in lxml.html would interpret CSS like url("link") incorrectly (treating the quotation marks as part of the link). * Failing import on systems that have an io module. 2.2beta1 (2008-12-12) Features added * Allow lxml.html.diff.htmldiff to accept Element objects, not just HTML strings. Bugs fixed * Crash when using an XPath evaluator in multiple threads. * Fixed missing whitespace before Link:... in lxml.html.diff. Other changes * Export lxml.html.parse. 2.2alpha1 (2008-11-23) Features added * Support for XSLT result tree fragments in XPath/XSLT extension functions. * QName objects have new properties namespace and localname. * New options for exclusive C14N and C14N without comments. * Instantiating a custom Element classes creates a new Element. Bugs fixed * XSLT didn't inherit the parse options of the input document. * 0-bytes could slip through the API when used inside of Unicode strings. * With lxml.html.clean.autolink, links with balanced parenthesis, that end in a parenthesis, will be linked in their entirety (typical with Wikipedia links).

1 0

[lxml-dev] lxml 2.2beta4 - release candidate for 2.2 final
by Stefan Behnel 20 Mar '09

20 Mar '09

Hi all, here is another almost-final version of lxml 2.2. Call it a release candidate, if you prefer. This release was necessary as the changelog was getting way too long, and the (crash-)bugs that were fixed in this release were too important to wait. So, updating is recommended. http://codespeak.net/lxml/dev/ http://pypi.python.org/pypi/lxml/2.2beta4 I'm very happy to announce that this is the first release that fully supports Python 3. The previous releases suffered from an annoying crash bug in Cython that was uncovered by the exception handling changes in Py3. This means that lxml now supports any officially released Python version from 2.3.x through 3.0.1. Note that the PDF documentation was broken in the last betas due to a recent system update on my side. It's back up for beta4. This release was built with revision "1796:cb0f315bb4f5" of the Cython 0.11 development branch. Have fun, Stefan 2.2beta4 (2009-02-27) Features added * Support strings and instantiable Element classes as child arguments to the constructor of custom Element classes. * GZip compression support for serialisation to files and file-like objects. Bugs fixed * Deep-copying an ElementTree copied neither its sibling PIs and comments nor its internal/external DTD subsets. * Soupparser failed on broken attributes without values. * Crash in XSLT when overwriting an already defined attribute using xsl:attribute. * Crash bug in exception handling code under Python 3. This was due to a problem in Cython, not lxml itself. * lxml.html.FormElement._name() failed for non top-level forms. * TAG special attribute in constructor of custom Element classes was evaluated incorrectly. Other changes * Official support for Python 3.0.1. * Element.findtext() now returns an empty string instead of None for Elements without text content.

2 1

[lxml-dev] Possible bug
by Bob Kline 20 Mar '09

20 Mar '09

Before I dig into the work of producing a repro case, would the lxml developers be interested in a bug report if I confirm that the XSL/T parser which comes with the lxml package chokes on the serialized version of an XML tree assembled by the lxml's HTML parser when the original HTML document contains a comment which the XML spec doesn't like (because "--" appears inside the comment)? -- Bob Kline http://www.rksystems.com mailto:bkline@rksystems.com

3 8

[lxml-dev] iterparse and namespaces
by Piotr Furman 17 Mar '09

17 Mar '09

Hi, I've got small problem with iterparse and namespaces. I have this code: >>> from StringIO import StringIO >>> from lxml import etree >>> print etree.__version__ 2.1.5 >>> xml = """<root xmlns="http://www.example.com"><a>1</a><a>2</a></root>""" >>> a1 = etree.iterparse(StringIO(xml), tag="a") >>> a1.next() Traceback (most recent call last): File "<stdin>", line 1, in ? File "iterparse.pxi", line 515, in lxml.etree.iterparse.__next__ (src/lxml/lxml.etree.c:77014) StopIteration >>> a2 = etree.iterparse(StringIO(xml), tag="{http://www.example.com}a") >>> a2.next() (u'end', <Element {http://www.example.com}a at -488456c4>) Is that possible to get all tags "a" without passing namespace? I mean, a2 works, but could it be possible to make a1 working too? regards Piotr Furman

6 11

[lxml-dev] space in attribute name: xpath expression?
by TP 17 Mar '09

17 Mar '09

Hi everybody, It seems not possible to define with fromstring() or ET.XML a tree containing attributes with spaces. But it is possible by adding the attribute containing a space afterwards, see the example below. ################### #!/usr/bin/env python # -*- coding: utf-8 -*- import lxml.etree as ET root = ET.XML("<root><foo attri='bar'>data</foo></root>") foo_elem = root.xpath( "//foo" ) foo_elem[0].set( "tu tu", "22" ) print ET.tostring( root ) ################### We obtain: <root><foo attri="bar" tu tu="22">data</foo></root> It seems a bad idea to have spaces in attributes. I have not found a way to make an xpath request work, for example the two following ones yield an error: print root.xpath( "//*[@tu tu=22]" ) print root.xpath( "//*[@tu\ tu=22]" ) >From another point of view, often we would like to define attribute names as they are, i.e. english expressions with spaces. How do you proceed? Put underscores in the attribute names, and then remove them when displaying in the tree (for example in a graphical widget)? Or define the correspondance between the attribute names and the english names in some part of the XML file (for example, the attribute names could be tags, associated to some text that would contain the english names. Thanks -- python -c "print ''.join([chr(154 - ord(c)) for c in '*9(9&(18%.\ 9&1+,\'Z4(55l4('])" "When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong." (first law of AC Clarke)

3 3

Re: [lxml-dev] the subelements of my tree are moving alone
by Stefan Behnel 14 Mar '09

14 Mar '09

[and to the list] TP wrote: > I have derived custom classes from ET._ElementTree and ET.ElementBase to > obtain a custom tree suited to my needs. It obviously depends on what you do with it, but I'm not so sure that subclassing the _ElementTree class is a good idea. It's not wrong in general, but maybe writing your own tree wrapper would be better. > It works perfectly, but it seems that the nodes under the root node (the > subelements) move sometimes "alone". The tree structure is kept, but the > address of the elements in memory is changing. As the structure is kept, it > is not a problem for lxml use only: I can walk in the tree, doing what I > need. > > But the problem is that I use this custom tree as the underlying data > structure for a PyQt custom QTreeWidget. In this widget, I use the > method "internalPointer()" of QModelIndex instances (as proposed in the > chapter 16 of book "Rapid GUI Programming with Python > and Qt" by Mark Summerfield (around p.500)). > > The problem is that if the nodes move, the "internalPointer()" of Qt are not > up to date: I obtain segmentation faults. > > Is this normal that nodes of the tree move in memory *alone*? Is this due to > the garbage collector? If yes, how to keep my pointers up to date? Here's some documentation: http://codespeak.net/lxml/dev/element_classes.html#background-on-element-pr… It's not only the "pointer", it's the Element instances that are replaced. It may be that "internalPointer()" (never heard of it, but I assume it's some kind of backpointing mechanism) does not respect Python's reference counting, so that the Element object gets garbage collected. A simple way to work around this is to keep a reference to all Element objects in the tree, initialised as my_tree.element_cache = list(root_element.iter()) But keep in mind that this needs to be maintained on tree changes. Stefan

1 0

Re: [lxml-dev] Fun With Sequencing
by Stefan Behnel 14 Mar '09

14 Mar '09

[and to the list] Hi, Tennis Smith wrote: > I am generating xml files (for test purposes) that will be validated > with a schema which uses "xs:sequence". The result is that my docs have > to have exactly the right sequence of elements or it will not validate. Could you say a bit more about your constraints? Why can't you just change the schema, if it's purely for test purposes? And why can't you just look at the schema once and generate the XML documents accordingly? > Its going to be a pain to re-parse the schema file and then use the > munged data to sort my xml docs. So, is there some way in lxml I can > dump the schema's contents and use that to make sure I have everything > in order *before* I try to validate? No, and I doubt that there's a major use case for that. Validation is about figuring out that something is wrong (any usually also what is wrong), not about getting stuff fixed automatically. Stefan

1 0

[lxml-dev] Fun With Sequencing
by Tennis Smith 11 Mar '09

11 Mar '09

Hi, I am generating xml files (for test purposes) that will be validated with a schema which uses "xs:sequence". The result is that my docs have to have exactly the right sequence of elements or it will not validate. Its going to be a pain to re-parse the schema file and then use the munged data to sort my xml docs. So, is there some way in lxml I can dump the schema's contents and use that to make sure I have everything in order *before* I try to validate? Failing that, are there any suggestions? Thanks -T

1 0