Mailman 3 February 2012 - lxml - The Python XML Toolkit

Trying to understand remove_blank_text
by Frank Millman 20 Feb '12

20 Feb '12

Hi all This is not of great importance, but I would like to understand the parser option 'remove_blank_text' better. I am assuming that 'blank text' refers to text that contains only whitespace. Python 3.2.2 (default, Sep 4 2011, 09:51:08) [MSC v.1500 32 bit (Intel)] on win32 >>> from lxml import etree >>> etree.__version__ '2.3.3' >>> parser = etree.XMLParser(remove_comments=True, remove_blank_text=True) >>> test = '<root> <node1> </node1> </root>' >>> testx = etree.fromstring(test, parser=parser) >>> etree.tostring(testx) b'<root><node1> </node1></root>' >>> I was expecting to see '<root><node1></node1></root>' It seems there is a difference between an element that contains only text, and an element that contains text plus another element. In the first case, blank text is not discarded, in the second case it is. >>> test = '<root><node1>  </node1></root>' >>> testx = etree.fromstring(test, parser=parser) >>> etree.tostring(testx) b'<root><node1> </node1></root>' >>> In this case, it removes the comment, and it removes one of the spaces around the comment. With further testing, I established that it removes the space before the comment, but not the one after. The reason that this cropped up is that I am editing an xml file by hand, and formatting it in a 'pretty-print' fashion. My program then uses etree.parse to read in the file and create a tree, using 'remove_comments' and 'remove_blank_text'. It works fine, but when I printed out a section for debugging, I was surprised to see that most whitespace is stripped, but this - <node>  </node> is stored as '<node>\n </node>', and I was curious to find out why. Can someone explain the rationale behind these results. Thanks Frank Millman

2 2

unicode behavior in child.text elements
by Piotr Oh 16 Feb '12

16 Feb '12

Hi I'm reading csv file encoded in cp1250 using csv python module. Then translate it to xml. This is the code I use .... for i in range(len(row)): child=etree.SubElement(jednostka, csvHeaders[i]) child.text=unicode(row[csvHeaders[i]].strip(), 'cp1250') print type(child.tag), child.tag, type(child.text), child.text .... the national characters can appear in some child.text but not in all of them. It depends on the data. But generally all the child.text should be encoded to unicode (?), but it is not the case. Only the data with national characters are encoded, the rest is of type str. Why? Is lxml selective in that case? But that looks strange, example: <a> <b>this is english</b> </a> <a> <b>źdźbło - polish</b> </a> and the lxml type representation of the elements looks like this: all tags are <str> and it is ok but the 'this is english' text is of type <str> and 'źdźbło - polish' is of type <unicode>. Is it normal? Finally, after serialization to xml, utf-8 encoded file looks ok, national characters are ok etc, so maybe it is not a problem but anyway I'm curious what is going on. P.

2 1

Cython errors building from git on Mac OS X
by Laurence Rowe 15 Feb '12

15 Feb '12

I'm having trouble building lxml from a git clone on Mac OS X 10.6 with python.org python 2.7 or my own compiled python 2.6. Any idea what might be causing the errors below? Thanks in advance, Laurence $ python2.7 setup.py build Building lxml version 2.4.dev. Building with Cython 0.15.1. Using build configuration of libxslt 1.1.24 Building against libxml2/libxslt in the following directory: /usr/local/lib running build running build_py creating build/lib.macosx-10.6-intel-2.7 creating build/lib.macosx-10.6-intel-2.7/lxml copying src/lxml/__init__.py -> build/lib.macosx-10.6-intel-2.7/lxml copying src/lxml/_elementpath.py -> build/lib.macosx-10.6-intel-2.7/lxml copying src/lxml/builder.py -> build/lib.macosx-10.6-intel-2.7/lxml copying src/lxml/cssselect.py -> build/lib.macosx-10.6-intel-2.7/lxml copying src/lxml/doctestcompare.py -> build/lib.macosx-10.6-intel-2.7/lxml copying src/lxml/ElementInclude.py -> build/lib.macosx-10.6-intel-2.7/lxml copying src/lxml/pyclasslookup.py -> build/lib.macosx-10.6-intel-2.7/lxml copying src/lxml/sax.py -> build/lib.macosx-10.6-intel-2.7/lxml copying src/lxml/usedoctest.py -> build/lib.macosx-10.6-intel-2.7/lxml creating build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/__init__.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/_dictmixin.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/_diffcommand.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/_html5builder.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/_setmixin.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/builder.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/clean.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/defs.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/diff.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/ElementSoup.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/formfill.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/html5parser.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/soupparser.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/usedoctest.py -> build/lib.macosx-10.6-intel-2.7/lxml/html creating build/lib.macosx-10.6-intel-2.7/lxml/isoschematron copying src/lxml/isoschematron/__init__.py -> build/lib.macosx-10.6-intel-2.7/lxml/isoschematron copying src/lxml/etreepublic.pxd -> build/lib.macosx-10.6-intel-2.7/lxml copying src/lxml/tree.pxd -> build/lib.macosx-10.6-intel-2.7/lxml copying src/lxml/etree_defs.h -> build/lib.macosx-10.6-intel-2.7/lxml creating build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources creating build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources/rng copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources/rng creating build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources/xsl copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources/xsl copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources/xsl creating build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 running build_ext cythoning src/lxml/lxml.etree.pyx to src/lxml/lxml.etree.c Error compiling Cython file: ------------------------------------------------------------ ... cdef _Element _node cdef _node_to_node_function _next_element cdef _MultiTagMatcher _matcher @cython.final cdef _initTagMatcher(self, tags): ^ ------------------------------------------------------------ src/lxml/lxml.etree.pyx:2501:9: The final compiler directive is not allowed in function scope Error compiling Cython file: ------------------------------------------------------------ ... def __iter__(self): return self @cython.final cdef int _storeNext(self, _Element node) except -1: ^ ------------------------------------------------------------ src/lxml/lxml.etree.pyx:2508:9: The final compiler directive is not allowed in function scope Error compiling Cython file: ------------------------------------------------------------ ... ] cimport cython cimport tree, python, config from tree cimport xmlDoc, xmlNode, xmlAttr, xmlNs, _isElement, _getNs ^ ------------------------------------------------------------ src/lxml/lxml.etree.pyx:43:0: 'xmlDoc.pxd' not found Error compiling Cython file: ------------------------------------------------------------ ... ] # snipped hundreds of similar errors.

3 3

[Pull 30] Use regexp for cssselect ":contains()" pseudo-class
by Laurence Rowe 14 Feb '12

14 Feb '12

This provides: * portability with other xslt processors * removes the 'css' global prefix assignment which can cause unexpected behaviour when working with xml files using that prefix * a slight performance improvement Users now need to explicitly pass in the regexp namespace when calling the ``xpath()`` method if they use the deprecated ``:contains()`` pseudo-class. https://github.com/lxml/lxml/pull/30 Laurence

1 1

External Entities in a Dictionary
by Klaus Schilling 12 Feb '12

12 Feb '12

Hello lxmlers, is it possible to pass a dictionary object to an etree.parser object for the lookup of external entities? Klaus Schilling

2 1

schema problem
by Marcin Krol 08 Feb '12

08 Feb '12

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello everyone, I have some (legacy == unmodifiable) xml files to handle and they cause problems with lxml. I'm not even sure if this node is correct: <test xmlns="TestAutomation" xmlns:xs="http://www.w3.org/2001/XMLSchema-instance" xs:schemaLocation="TestAutomation TestAutomation.xsd" > The problem is that when parsing file with such (root) node, I get this in namespace: root.nsmap {'xs': 'http://www.w3.org/2001/XMLSchema-instance', None: 'TestAutomation'} This makes the tree unparseable (iterators don't work, xpath doesn't work), specifically, None key causes this problem - even if I pass root.nsmap to 'namespaces' in element.xpath() call, I get TypeError (if I pass a copy of dictionary without None key, xpath and iterators silently fail, that is, they return empty results). OTOH, this element is impossible to delete from the namespace - that is: del root.nsmap[None] executes, but the "None" key is still there in the root.nsmap dictionary. Is this a bug or is this incorrect node schema attribs? Regards, Marcin Krol -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJPMnljAAoJEFMgHzhQQ7hOBwEH/RupqdW1TRB+xEnZulZ4a39D QawtgNQTmCN+WiOJ6PHTtJn2tr6bdEKbYL+QYKVCc6uwULnjy+e/ekW2NmN4iSiS gEMz4FmIrTXD6y1cgZr3Vj349rR7EBZM1Dgg11nDVZHF7z+8qWt91mfI+G4MDAY/ 6RM3FoBPmm+I+fHsFgxZvJjnnFFX/vNnLCvwa8qy1dPQLjYAby1wZhVGCy7dt1dk c3RDqSLNrbKOxy2IDg5/NbEovpewgjS/cJvjxczmIUn3Q+iBzjtlPqfLt+F1fZ5c 5nL25lPWWVx9+OWT+3/eTbvGKUYYHmCS5rz6+5ef7BIePHSqjeHVmi/9mB4wuU0= =tyEV -----END PGP SIGNATURE-----

3 2

schema problem
by Marcin Krol 08 Feb '12

08 Feb '12

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Argh, I forgot to include info: Python : sys.version_info(major=2, minor=7, micro=2, releaselevel='final', serial=0) lxml.etree : (2, 3, 3, 0) libxml used : (2, 7, 8) libxml compiled : (2, 7, 8) libxslt used : (1, 1, 26) libxslt compiled : (1, 1, 26) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJPMnnWAAoJEFMgHzhQQ7hOEA8H/AsO7oxH87nY1FvXufRA9kKM dSXfM5oVcOw8IrtI6mv19fpWHL+UrBtyWwxHq6q5VpT9uvH1bq5mjdBcDjIbBEo1 wXdd4aReqSIJLRuSwkNI43SuWo4G/oR3LBUBoBrSHGYD8we92S4d+7paaS0A9SGZ t6CSGgkuMozpEd2k82yS6JNqvZvyQEDdKpeJdFkxdsEk+EOy5l5aD7+uIoFnbEXK tgxaQ8VBdidw7cLPlD7MqW1EZOaIPrew+WQSZ4AtJGeRlTRec6uTfONwrKWKNo8+ BrnM0MWujyMuFcqonX5VoyBInwiYFfYTZQVqZQNzqZFnXAqMTc2Zvw8eDfAu1JI= =Vo1G -----END PGP SIGNATURE-----

1 0

Re: [lxml] Allowing bad characters
by David Roe 07 Feb '12

07 Feb '12

> David Roe, 06.02.2012 18:48: > > I have to load an XML file generated by a third-party program. They > > have some non-printing chars in the document and I get a > > "lxml.etree.XMLSyntaxError: xmlParseCharRef: invalid xmlChar value..." > > exception. > > Looks like it's not giving you XML then. Could you post an example of the character references that it cannot parse? Here are the ones:  Obviously those aren't valid XML characters (hence my orignial post). Unfortunately this is the reality of the file I need to load. The way I see it I have a few options: 1) Find some way to get lxml to load those. 2) String replace them with some placeholder before parsing. 3) Roll my own XML (almost) parser. I'd obviously rather not do (2) or (3) which is why I was asking if there is any way to load this file with lxml. I take it from your reply that (1) is not a possibility? > But please take care to send your next post without the legal restrictions at the bottom of your first e-mail. I can't switch that off so I'm sending from another account. > Hmm, I can't find myself in the list of recipients - am I authorised to keep a copy of your e-mail or not? Sorry for not having my phone within reach while receiving e-mails. You are a member of the mailing list I sent it to so yes you are.

1 0

Allowing bad characters
by David Roe 07 Feb '12

07 Feb '12

I have to load an XML file generated by a third-party program. They have some non-printing chars in the document and I get a "lxml.etree.XMLSyntaxError: xmlParseCharRef: invalid xmlChar value..." exception. Is there any way to relax or switch off the allowed characters in lxml? If not then is there a way to plug into the character entity parsing code? I really don't want to resort to a pre-parse regex replace. Thanks This E-Mail is sent in confidence for the addressee only. Unauthorised recipients must preserve this confidentiality and should please advise the sender immediately by telephone (+44 (0)1625 505100) and return the original E-Mail to the sender without taking a copy. Cyprotex has taken all reasonable precautions to ensure that no viruses are transmitted from Cyprotex to any third party. Cyprotex accepts no responsibility for any loss or damage resulting directly or indirectly from the use of this E-Mail or the contents.

2 1